emc business continuity for oracle database 11g/10g · best practices ... emc business continuity...

EMC Business Continuity for Oracle Database 11g/10g

Enabled by EMC CLARiiON CX4 and

EMC Celerra Using FCP and NFS

Proven Solution Guide

Copyright © 2010 EMC Corporation. All rights reserved. Published January 2010 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. Benchmark results are highly dependent upon workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, this workload should not be used as a substitute for a specific customer application benchmark when critical capacity planning and/or product evaluation decisions are contemplated. All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute. No warranty of system performance or price/performance is expressed or implied in this document. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners. Part number: H6647

EMC Business Continuity for Oracle Database 11g/10g Enabled by EMC CLARiiON CX4 and EMC Celerra

Using FCP and NFS Proven Solution Guide 3

Table of Contents

Chapter 1: About this Document ............................................................................................................... 6

Overview ........................................................................................................................................... 6 Audience and purpose ...................................................................................................................... 7 Scope ................................................................................................................................................ 8 Business challenge ......................................................................................................................... 10 Technology solution ........................................................................................................................ 10 Reference Architecture ................................................................................................................... 14 Validated environment profile.......................................................................................................... 15 Hardware and software resources .................................................................................................. 15 Prerequisites and supporting documentation .................................................................................. 17 Terminology..................................................................................................................................... 18 Typographic conventions ................................................................................................................ 20

Chapter 2: Storage Design...................................................................................................................... 21 Overview ......................................................................................................................................... 21 Concepts ......................................................................................................................................... 22 Best practices .................................................................................................................................. 23 CX4 cache configuration for SnapView snapshot ........................................................................... 25 Storage processor failover .............................................................................................................. 25 LUN/RAID group layout ................................................................................................................... 26 Storage design layout ..................................................................................................................... 28

Chapter 3: File System ............................................................................................................................ 30 Overview ......................................................................................................................................... 30 File system layout ........................................................................................................................... 31

Chapter 4: Application Design ................................................................................................................ 32 Overview ......................................................................................................................................... 32 Considerations ................................................................................................................................ 33 Application design layout ................................................................................................................ 34 Memory configuration for Oracle 11g .............................................................................................. 35 HugePages...................................................................................................................................... 37

Chapter 5: Network Design ..................................................................................................................... 38 Overview ......................................................................................................................................... 38 Concepts ......................................................................................................................................... 39 Best practices .................................................................................................................................. 40 SAN network layout ......................................................................................................................... 41 IP network layout ............................................................................................................................. 41 Virtual LANs .................................................................................................................................... 41 Jumbo frames.................................................................................................................................. 42 Ethernet trunking and link aggregation ........................................................................................... 43 Public and private networks ............................................................................................................ 45 Oracle RAC 11g/10g server network architecture .......................................................................... 45



Chapter 6: Installation and Configuration ................................................................................................ 47 Overview ......................................................................................................................................... 47 Task 1: Build the network infrastructure .......................................................................................... 48 Task 2: Set up and configure ASM for CLARiiON .......................................................................... 48 Task 3: Set up and configure database servers ............................................................................. 49 Task 4: Configure NFS client options ............................................................................................. 50 Task 5: Install Oracle Database 11g/10g ........................................................................................ 51 Task 6: Configure database server memory options ...................................................................... 51 Task 7: Tune HugePages ............................................................................................................... 54 Task 8: Set database initialization parameters ............................................................................... 56 Task 9: Configure Oracle Database control files and logfiles ......................................................... 58 Task 10: Enable passwordless authentication using SSH .............................................................. 59 Task 11: Set up and configure CLARiiON storage for Replication Manager and SnapView ......... 64 Task 12: Install and configure EMC RecoverPoint ......................................................................... 67 Task 13: Set up the virtualized utility servers .................................................................................. 70 Task 14: Configure and connect EMC RecoverPoint appliances (RPAs) ...................................... 71 Task 15: Install and configure EMC MirrorView/A .......................................................................... 71 Task 16: Install and configure EMC CLARiiON (CX) splitters ........................................................ 71

Chapter 7: Testing and Validation ........................................................................................................... 72 Overview ......................................................................................................................................... 72

Section A: Store solution component .................................................................................................. 74 Test results ..................................................................................................................................... 75

Section B: Basic Backup solution component ..................................................................................... 79 Test results ...................................................................................................................................... 80

Section C: Advanced Backup solution component ............................................................................. 81 Test results ...................................................................................................................................... 82

Section D: Basic Protect solution component ..................................................................................... 84 Test results ...................................................................................................................................... 91

Section E: Advanced Protect solution component using EMC MirrorView and Oracle Data Guard .. 92 Test results ..................................................................................................................................... 93

Section F: Advanced Protect solution component using EMC RecoverPoint ..................................... 95 Test results ..................................................................................................................................... 97

Section G: Test/Dev solution component using EMC SnapView clone .............................................. 98 Test results .................................................................................................................................... 100

Section H: Backup Server solution component ................................................................................. 101 Test results .................................................................................................................................... 102

Section I: Migration solution component ........................................................................................... 103 Test results .................................................................................................................................... 104

Chapter 8: Virtualization ........................................................................................................................ 105 Overview ....................................................................................................................................... 105 Advantages of virtualization .......................................................................................................... 106 Considerations .............................................................................................................................. 106 VMware infrastructure ................................................................................................................... 106



Virtualization best practices .......................................................................................................... 106 VMware ESX server ...................................................................................................................... 107 VMware and NFS .......................................................................................................................... 108

Chapter 9: Backup and Restore ............................................................................................................ 109 Overview ....................................................................................................................................... 109

Section A: Backup and restore concepts .......................................................................................... 110 Advantages of logical storage ....................................................................................................... 111

Section B: Backup and recovery strategy ......................................................................................... 112 Logical storage backup using EMC SnapView and EMC Replication Manager ........................... 113

Section C: Physical backup and restore ........................................................................................... 116 Physical backup using Oracle RMAN ........................................................................................... 116

Section D: Replication Manager in Test/Dev and Advanced Backup solution components ............. 119 Chapter 10: Data Protection and Replication........................................................................................ 120

Overview ....................................................................................................................................... 120 Section A: Basic Protect using Oracle Data Guard........................................................................... 121 Section B: Advanced Protect using EMC MirrorView and Oracle Data Guard ................................. 122 Section C: Advanced Protect using EMC RecoverPoint ................................................................... 127

Overview ....................................................................................................................................... 127 EMC RecoverPoint ........................................................................................................................ 128 CLARiiON (CX) splitters ................................................................................................................ 128 Scope ............................................................................................................................................ 129 Best practices ................................................................................................................................ 129 Conclusion ..................................................................................................................................... 130

Chapter 11: Test/Dev Solution Using EMC SnapView Clone ............................................................... 131 Overview ....................................................................................................................................... 131 CLARiiON SnapView clone ........................................................................................................... 132 Best practices ................................................................................................................................ 132 Mount and recovery of a target clone database using Replication Manager ................................ 133 Database cloning .......................................................................................................................... 136

Chapter 12: Migration ............................................................................................................................ 137 Overview ....................................................................................................................................... 137

Chapter 13: Conclusion ......................................................................................................................... 139 Overview ....................................................................................................................................... 139

Chapter 1: About this Document


using FCP and NFS Proven Solution Guide 6


Overview

Introduction EMC's commitment to consistently maintain and improve quality is led by the Total Customer Experience (TCE) program, which is driven by Six Sigma methodologies. As a result, EMC has built Customer Integration Labs in its Global Solutions Centers to reflect real-world deployments in which TCE use cases are developed and executed. These use cases provide EMC with an insight into the challenges currently facing its customers. The introduction of the EMC® Celerra® unified storage platform prompted the creation of a solution that showcases the new capability of this unit: to expose the back-end EMC CLARiiON® array to hosts for Fibre Channel Protocol (FCP) access, in addition to the normal Network File System (NFS) access previously provided by the Celerra NS Series. In this solution, all Oracle objects that require higher performance and lower latency I/O are placed over an FCP connection, and all other objects are placed over an NFS connection. This may sound counterintuitive as it requires the management of two separate storage protocols. However, it is far simpler to manage and configure this solution than to manage a solution using the FCP alone, and this solution provides identical performance. Thus, management and simplicity are improved by the blended FCP/NFS solution, while performance is not affected. This document summarizes a series of implementation procedures and best practices that were discovered, validated, or otherwise encountered during the validation of a solution for Oracle Database 11g/10g using the EMC Celerra unified storage platform and Oracle RAC 11g and 10g on Linux over FCP and NFS.

Contents The content of this chapter includes the following topics.

Topic See Page

Audience and purpose 7

Scope 8

Business challenge 10

Technology solution 10

Reference Architecture 14

Validated environment profile 15

Hardware and software resources 15

Prerequisites and supporting documentation 17

Terminology 18

Typographic conventions 20




Audience and purpose

Audience The intended audience for the Proven Solution Guide is:

• Internal EMC personnel • EMC partners • Customers

Purpose The purpose of this solution is to:

• Improve the performance, scalability, flexibility, and resiliency of an Oracle

software stack that is physically booted on normal hardware by connecting multiple protocols to one storage platform as follows: − Fibre Channel Protocol (FCP) and Oracle ASM to access high-demand, low-

latency storage elements − NFS to access all other storage elements

• Facilitate and reduce the risk of migrating an existing Oracle Database 10g installations to 11g by providing documentation of best practices.

• Reduce cost by migrating an online production Oracle Database mounted over FCP to a target database mounted over NFS with no downtime and minimal performance impact. Improve the performance of an Oracle 11g or 10g production database by offloading all performance impacts of database operations, such as backup and recovery, using: − EMC Replication Manager − EMC SnapViewTM These demonstrate significant performance and manageability benefits in comparison to normal Oracle Recovery Manager (RMAN) backup and recovery.

• Provide disaster recovery capability using: − EMC RecoverPoint with CLARiiON splitters − EMC MirrorViewTM/Asynchronous over iSCSI − These demonstrate significant performance and manageability benefits in

comparison to normal Oracle Data Guard disaster recovery. • Provide the capability to clone a running production database with minimal

performance impact and no downtime using SnapView clones and Replication Manager.




Scope

Overview This section describes the components of the solution.

Core solution components

The following table describes the core solution components that are included in this solution:

Component Description

Scale-up OLTP Using an industry-standard OLTP benchmark against a single database instance, comprehensive performance testing is performed to validate the maximum achievable performance using the solution stack of hardware and software.

Resiliency The purpose of resiliency testing is to validate the fault-tolerance and high-availability features of the hardware and software stack. Faults are inserted into the configuration at various layers in the solutions stack. Some of the layers where fault tolerance is tested include: Oracle RAC node, Oracle RAC node interconnect port, storage processors, and Data Movers.

Functionality solution components

The following table describes the functionality solution components that are included in this solution:

Component Description

Basic Backup This is backup and recovery using Oracle RMAN, the built-in backup and recovery tool provided by Oracle.

Advanced Backup

This is backup and recovery using EMC value-added software or hardware. In this solution, the following are used to provide Advanced Backup functionality: • EMC Replication Manager • EMC SnapView snapshot

Basic Protect This is disaster recovery using Oracle Data Guard, Oracle’s built-in remote replication tool.

Advanced Protect This is disaster recovery using EMC value-added software and hardware: In this solution the following are used to provide Advanced Protect functionality: • EMC RecoverPoint with CLARiiON splitters • EMC MirrorView/A over iSCSI

Test/dev A running production OLTP database is cloned with minimal, if any, performance impact on the production server, as well as no downtime. The resulting dataset is provisioned on




another server for use for testing and development. EMC Replication Manager is used to automate the test/dev process.

Migration An online production Oracle database that is mounted over FCP/ASM is migrated to a target database mounted using NFS, with no downtime and minimal performance impact on the production database.




Business challenge

Business challenges for midsize enterprises

Midsize enterprises face the same challenges as their larger counterparts when it comes to managing database environments. These challenges include: • Rising costs • Control over resource utilization and scaling • Lack of sufficient IT resources to deploy, manage, and maintain complex

environments at the departmental level • The need to reduce power, cooling, and space requirements Unlike large enterprises, midsize enterprises are constrained by smaller budgets and cannot afford a custom, one-off solution. This makes the process of creating a database solution for midsize enterprises even more challenging than for large enterprises.

Technology solution

Blended solution for midsize enterprises

This solution demonstrates how organizations can: • Deploy a solution using a combination of the NFS and FCP protocols on the

Celerra. − FCP is used for high-I/O and low-latency database objects (notably the datafiles,

tempfiles, online redo logfiles, and controlfiles). − NFS is used for all other database objects (consisting basically of the flashback

recovery area, disk-based backups, archive logs, and CRS files). − Manageability advantages are obtained by using a combination of FCP and

NFS. Specifically, archived logs and backups can be accessed through a normal file system interface rather than ASM. Further, another clustered file system is not required for the CRS files. This simplifies the software installation and configuration on the database servers.

• Avoid investing in additional FC infrastructure by implementing a blended solution that uses both FCP and NFS to access storage elements.

• Work with different protocols in the blended solution to migrate an online production Oracle Database mounted over FCP to a target database mounted over NFS, with no downtime and minimal performance impact on the production database.

• Maximize the use of the database-server CPU, memory, and I/O channels by offloading performance impacts from the production server during: − Backup and recovery by using Replication Manager or SnapView − Disaster recovery operations by using RecoverPoint or MirrorView/A

• Reduce the complexity of backup operations and eliminate the need to implement scripted solutions by using Replication Manager.

• Save time and maximize system uptime when migrating existing Oracle Database 10g systems to Oracle Database 11g.

• Implement a disaster recovery solution with MirrorView/A over iSCSI that reduces




costs and complexity by using IP as the network protocol. • Use SnapView to free up the database server’s CPU, memory, and I/O channels

from the effects of operations relating to backup, restore, and recovery. SnapView clones also help in creating test/development systems without any impact on the production environment.

Blended FCP/NFS solution

This is a blended FCP/NFS solution. Depending on the nature of the database object, either FCP or NFS is used to access it. The following table shows which protocol is used to access each database object.

Database object Type Accessed using

Datafiles

High demand Low latency

FCP Online redo logfiles

Controlfiles

Tempfiles

Flashback recovery area

Low demand High latency

NFS Archive logs

Disk-based backups

CRS files

Two sites connected by WAN

Two sites connected by a WAN are used in the solution: one site is used for production; the other site is used as a disaster recovery target. A Celerra is present at each site. Oracle RAC 11g or 10g for x86-64 is run on Red Hat Enterprise Linux or on Oracle Enterprise Linux. FCP storage networks consisting of dedicated, redundant FCP switches are present at both sites. An EMC RecoverPoint cluster is also included at each site. The solution includes virtualized servers for use as Test/dev, Basic Protect, and Advanced Protect targets. Virtualization of the test/dev and disaster recovery (DR) target servers is supported using VMware ESX Server.

Production site The following components are present at the production site and are connected to

the production FCP storage network and to the WAN: • A Celerra (actually the CLARiiON back-end array) • A physically booted four-node Oracle RAC 11g or 10g cluster • A RecoverPoint cluster connected to the FCP storage network and the WAN The Oracle RAC 11g or 10g servers are also connected to the client and RAC interconnect networks.




Disaster recovery target site

The disaster recovery target site consists of: • A target Celerra (actually the CLARiiON back-end array) connected to the target

FCP storage network • A RecoverPoint cluster connected to the FCP storage network and the WAN

Connected to both sites

The following are present at both sites: • A VMware ESX server is connected to both the production and target FCP storage

networks. • A virtualized single-instance Oracle 11g or 10g server is used as: − The disaster recovery target for Basic Protect and Advanced Protect (DR site) − The target for Test/Dev (production site) The virtualized single-instance Oracle 11g or 10g target server accesses both the production and target FCP storage networks and is connected to the client WAN through virtualized connections on the virtualization server.

• A virtualized Replication Manager Server is responsible for handling replication tasks through the Replication Manager Agent, which is installed on the production database servers.

The LUNs on the Celerra are discovered using Raw Device Mapping (RDM) on the target VMs.

Storage layout The following table describes how each Oracle file type and database object is

stored and accessed for this solution:

What Protocol Stored on File-system type

Oracle datafiles

FCP FC disk (LUNs)

RAID-protected ASM diskgroup

Oracle tempfiles

Oracle online redo logfiles

Oracle controlfiles

Voting disk

NFS

FC disk

RAID-protected NFS

OCR files

Archived logfiles

SATA II Flashback recovery area

Backup target High-performance database objects are accessed over an FCP network using

redundant network switches.




ASM and ASMLib

Oracle ASM is used as the file system/volume manager. Oracle ASMLib is used to virtualize the LUNs on the database server. • Oracle datafiles, tempfiles, and online redo logfiles are stored on separate LUNs

that are mounted on the database server using ASM over FCP. • Three ASM diskgroups are used — one diskgroup for datafiles and tempfiles, and

two diskgroups for online redo logfiles. • The online redo logfiles are mirrored across the two ASM diskgroups using Oracle

software multiplexing. The controlfiles are mirrored across the online redo log ASM diskgroups.

• Each ASM diskgroup and its underlying LUNs are designed to satisfy the I/O demands of individual database objects. For example, RAID 5 is used for the datafiles and the tempfiles, but RAID 1 is used for the online redo logfiles. All of these diskgroups are stored on FC disks.

Network architecture

TCP/IP and NFS provide network connectivity and file system semantics for NFS file systems on Oracle RAC 11g or 10g. • Client virtual machines run on the VMware ESX server. They are connected to a

client network. • Client, RAC interconnect, and redundant TCP/IP storage networks consist of

dedicated network switches and virtual local area networks (VLANs). • The RAC interconnect and storage networks consist of trunked IP connections to

balance and distribute network I/O. Jumbo frames are enabled on these networks.




Reference Architecture

Corresponding Reference Architecture

This solution has a corresponding Reference Architecture document that is available on Powerlink®, EMC.com, and EMC|KB.WIKI. Refer to EMC Business Continuity for Oracle Database 11g/10g - Enabled by EMC CLARiiON CX4 and EMC Celerra Using FCP and NFS Reference Architecture for details. If you do not have access to this content, contact your EMC representative.

Reference Architecture diagram

The following diagram depicts the overall physical architecture of the solution.




Validated environment profile

Environment profile and test results

For information on the performance results, refer to the testing summary results contained in Chapter 7: Testing and Validation.

Hardware and software resources

Hardware The hardware used to validate the solution is listed below. Equipment Quantity Configuration

EMC Celerra unified storage platforms (includes an EMC CLARiiON CX4 back-end storage array)

2 • 2 Data Movers • 4 GbE network connections per Data Mover • 2 or 3 FC shelves • 1 SATA shelf • 30 or 45 73 GB FC disks (depending on configuration) • 15 500 GB SATA disks • 1 Control Station • 2 storage processors • DART version 5.6.44-4

Dell PowerConnect Gigabit Ethernet switches

5 24 ports per switch

QLogic FCP switches 2 • 16 ports • 4 Gb throughput

Database servers Dell PowerEdge 2900 (Oracle RAC 11g/10g servers)

4 • 2 2.66 GHz Intel Pentium 4 quad-core processors • 24 GB of RAM • 2 146 GB 15k internal SCSI disks • 2 onboard GbE Ethernet NICs • 2 additional Intel PRO/1000 PT quad-port GbE Ethernet NICs • 2 SANblade QLE2462-E-SP 4 Gb/s dual-port FC HBAs

(4 ports in total)

EMC RecoverPoint appliances (RPA)

4 • 2 Dell 2950 servers per site • QLA2432 HBA cards

Virtualization server Dell PowerEdge 6450 (VMware ESX server)

1 • 4 2.86 GHz AMD Opteron quad-core processors • 32 GB of RAM • 2 146 GB 15k internal SCSI disks • 2 onboard GbE Ethernet NICs • 3 additional Intel PRO/1000 PT quad-port GbE Ethernet NICs • 2 SANblade QLE2462-E-SP 4 Gb/s dual-port FC HBAs

(4 ports in total)




Software The software used to validate the solution is listed below. Software Version

Oracle Enterprise Linux 4.7

VMware ESX Server/vSphere 4.0

Oracle VM 2.1.1

Microsoft Windows Server 2003 Standard Edition

2003

Oracle RAC Enterprise Edition

11g or 10g

Oracle Database Standard Edition

11g or 10g (11g version 11.1.0.7.0)

Quest Benchmark Factory for Databases 5.8.1

EMC Celerra Manager Advanced Edition

5.6

EMC Navisphere® Agent 6.26.0.2.24

EMC PowerPath® 5.0.0 (build 157)

EMC FLARE® 04.28.000.5.504

EMC DART 5.6.44-4

EMC Navisphere Management 6.28

EMC RecoverPoint 3.0 SP1

EMC Replication Manager 5.2.0.0

EMC MirrorView 6.7

EMC CLARiiON splitter driver 04.28.006.6.003




Prerequisites and supporting documentation

Technology It is assumed the reader has a general knowledge of:

• EMC Celerra • EMC CLARiiON CX4 • Oracle Database (including RMAN and Data Guard) • EMC SnapView • EMC Replication Manager • EMC RecoverPoint • EMC MirrorView • VMware ESX Server • VMware vSphere

Supporting documents

The following documents, located on Powerlink.com, provide additional, relevant information. Access to these documents is based on your login credentials. If you do not have access to the following content, contact your EMC representative. • CLARiiON CX4 series documentation • EMC Unified Storage for Oracle Database 11g/10g - Physically Booted Solution

Enabled by EMC Celerra and Linux using FCP and NFS Reference Architecture

Third-party documents

The following resources have more information about Oracle: • Oracle Technology Network • MetaLink • Oracle support

http://www.oracle.com/technology/index.html�

http://metalink.oracle.com/�

http://www.oracle-support.com/�




Terminology

Terms and definitions

This section defines the terms used in this document.

Term Definition

Solution A solution is a complete stack of hardware and software upon which a customer would choose to run their entire business or business function. A solution includes database server hardware and software, IP networks, storage networks, storage array hardware and software, among other components.

Core solution component

A core solution component addresses the entire solution stack, but does so in a way relating to a discrete area of testing. For example, performance testing is a core solution component.

Functionality solution component

A functionality solution component addresses a subset of the solution stack that consists of a discrete set of hardware or software, and focuses on a single IT function. For example, backup and recovery, and disaster recovery are both functionality solution components. A functionality solution component can be either “basic” or “advanced”.

Basic solution component

A basic solution component uses only the features and functionality provided by the Oracle stack. For example, RMAN is for backup and recovery, and Data is Guard for disaster recovery.

Advanced solution component

An advanced solution component uses the features and functionality of EMC hardware or software. For example, EMC SnapView is for backup and recovery, and EMC MirrorView is for disaster recovery.

Physically-booted solution

A configuration in which the production database servers are directly booted off a locally attached hard disk without the use of a hypervisor such as VMware or Oracle VM. Utility servers (such as test/dev target or disaster recovery target) may still be virtualized in a physically-booted solution.

Virtualized solution A configuration in which the production database servers are virtualized using a hypervisor technology such as VMware or Oracle VM.

Scale-up The use of a clustered or single-image database server configuration. Scaling is provided by increasing the number of CPUs in the database server (in the case of a single-instance configuration) or by adding nodes to the cluster (in the case of a clustered configuration). Scale-up assumes that all customers of the database will be able to access all database data.

Resiliency Testing that is designed to validate the ability of a configuration to withstand faults at various layers. The layers that are tested include: network switch, database server storage network port, storage array network port, database server cluster node, and storage processor.

Test/dev The use of storage layer replication (such as snapshots and clones) to provide




an instantaneous, writeable copy of a running production database with no downtime on the production database server and with minimal, if any, performance impact on the production server.

Advanced Backup and Recovery

A solution component that provides backup and recovery functionality through the storage layer using specialized hardware or software. Advanced Backup and Recovery has the following benefits: • Offloads the database server’s CPUs from the I/O and processing

requirements of the backup and recovery operations • Superior Mean Time to Recovery (MTTR) through the use of virtual storage

layer replication (commonly referred to as snapshots)

Basic Backup and Recovery

A solution component that provides backup and recovery functionality through the operating system and database server software stack. Basic Backup and Recovery uses the database server’s CPUs for all I/O and processing of backup and recovery operations.

Advanced Protect A solution component that provides disaster recovery functionality provided through the storage layer using specialized hardware or software. Advanced Protect has the following benefits: • Offloads the database server’s CPUs from the I/O and processing

requirements of the disaster recovery operations • Superior failover and failback capabilities • Reduces the software required to be installed at the disaster recovery target

because of the use of consistency technology

Basic Protect A solution component that provides disaster recovery functionality provided through the operating system and database server software stack. Basic Protect uses the database server’s CPUs for all I/O and processing of disaster recovery operations.

Kernel NFS (KNFS) A network storage protocol in which the NFS client is embedded in the operating system kernel.

High availability The use of specialized hardware or software technology to reduce both planned and unplanned downtime.

Fault tolerance The use of specialized hardware or software technology to eliminate both planned and unplanned downtime.

Enterprise Flash Drive (EFD)

A drive that stores data using Flash memory and contains no moving parts.

Serial advanced technology-attachment (SATA) drive

SATA is a newer standard for connecting hard drives into computer systems. SATA is based on serial signaling technology, while Integrated Drive Electronics (IDE) hard drives use parallel signaling.

Migration The ability to transfer a running production database from one environment to another, for example, from FCP/ASM to KNFS/DNFS.




Typographic conventions

Typographic conventions

In this document, many steps are listed in the form of terminal output. This is referred to as a code listing. For example:

Note the following about code listings: • Commands you type are shown in bold. • For lengthy commands the backslash “\” character is used to show line

continuation. While this is a common UNIX convention, it may not work in all cases. You should enter the command on one line.

• The use of ellipses (…) in the output indicates that lengthy output was deleted for brevity.

If a Celerra or Linux command is referred to in text it is indicated in bold and lowercase, like this: the fs_copy command. If a SQL or RMAN command is referred to in text, it is indicated in uppercase, like this: The ALTER DATABASE RENAME FILE command. A special font is not used in either case.

Chapter 2: Storage Design




Overview

Introduction to Storage Design

The storage design layout instructions presented in this chapter apply to the specific components used during the development of this solution.

Contents This chapter contains the following topics:

Topic See Page

Concepts 22

Best practices 23

CX4 cache configuration for SnapView snapshot 25

Storage processor failover 25

LUN/RAID group layout 26

Storage design layout 28




Concepts

Setting up CX storage

To set up CLARiiON (CX) the following steps must be carried out:

Step Action

1 Configure zoning.

2 Configure RAID groups and bind LUNs.

3 Allocate hot spares.

4 Create storage groups.

5 Discover FCP LUNs from the database servers.

High availability and failover

EMC Celerra has built in high-availability (HA) features. These HA features allow the Celerra to survive various failures without a loss of access to the Oracle database. These HA features protect against the following: • Power loss affecting a single circuit connected to the storage array • Storage processor failure • Storage processor reboot • Disk failure




Best practices

Disk drives The following are the general recommendations for disk drives:

• Drives with higher revolutions per minute (rpm) provide higher overall random-

access throughput and shorter response times than drives with slower rpm. For optimum performance, higher-rpm drives are recommended for datafiles and tempfiles as well as online redo logfiles.

• Because of significantly better performance, Fibre Channel drives are always recommended for storing datafiles, tempfiles, and online redo log files.

• Serial Advanced Technology-Attached (SATA II) drives have slower response and rotational speed, and moderate performance with random I/O. However, they are less expensive than the Fibre Channel drives for the same or similar capacity.

SATA II drives are frequently the best option for storing archived redo logs and the flashback recovery area. In the event of high performance requirements for backup and recovery, Fibre Channel drives can also be used for this purpose.

Enterprise Flash Drives (EFDs)

Enterprise Flash Drives (EFDs) can be used to dramatically improve cost, performance, efficiency, power, space, and cooling requirements of Oracle databases stored on EMC Celerra. To know if EFDs will fit in your situation, you need to determine if a set of datafiles is being accessed more heavily than other datafiles. This is an extremely common condition in Oracle databases. If so, then migrate this set of datafiles to EFDs. The datafiles that are affected may change over time, requiring application of an Information Lifecycle Management (ILM) strategy. Because of the sequential nature of redo I/O, we do not recommend storing online redo logfiles on EFDs. EFDs primarily accelerate random read and write I/O.

RAID types and file types

The following table describes the recommendations for RAID types corresponding to Oracle file types:

Description RAID 5/EFD RAID 10/FC RAID 5/FC RAID 5/SATA II

Datafiles/tempfiles Possible (apply tuning) 1

Recommended Recommended Avoid

Control files Avoid Recommended Recommended Avoid

Online redo logs Avoid Recommended Avoid Avoid

Archived logs Avoid Possible (apply tuning) 1

Possible (apply tuning) 2

Recommended

Flashback recovery area

Avoid OK OK Recommended




OCR file/voting disk

Avoid OK OK Avoid

1 The decision to use EFDs for datafiles and tempfiles must be driven by the I/O requirements for specific datafiles. 2 The use of FC disks for archived logs is fairly rare. However, if many archived logs are being created, and the I/O requirements for archived logs exceeds a reasonable number of SATA II disks, this may be a more cost-effective solution.

Tempfiles, undo, and sequential table or index scans

In some cases, if an application creates a large amount of temp activity, placing your tempfiles on RAID 10 devices may be faster due to RAID 10’s superior sequential I/O performance. This is also true for undo. Further, an application that performs many full table scans or index scans may benefit from these datafiles being placed on a separate RAID 10 device.

Online redo logfiles

Online redo log files should be put on RAID 1 or RAID 10 devices. You should not use RAID 5 because sequential write performance of distributed parity (RAID 5) is not as high as that of mirroring (RAID 1). RAID 1 or RAID 10 provides the best data protection; protection of online redo log files is critical for Oracle recoverability.

OCR files and voting disk files

You should use FC disks for OCR files and voting disk files; unavailability of these files for any significant period of time (due to disk I/O performance issues) may cause one or more of the RAC nodes to reboot and fence itself off from the cluster. The LUN/RAID group layout images in Chapter 2: Storage Design > LUN/RAID group layout, show two different storage configurations that can be used for Oracle RAC 11g/10g databases on a Celerra. That section can help you to determine the best configuration to meet your performance needs.

Stripe size EMC recommends a stripe size of 32 KB for all types of database workloads.

The default stripe size for all the file systems on FC shelves (redo logs and data) should be 32 KB. Similarly, the recommended stripe size for the file systems on SATA II shelves (archive and flash) should be 256 KB.

Shelf configuration

The most common error when planning storage is designing for capacity rather than for performance. The single most important storage parameter for performance is disk latency. High disk latency is synonymous with slower performance; low disk counts lead to increased disk latency. The recommendation is a configuration that produces average database I/O latency (the Oracle measurement db file sequential read) of less than or equal to 20 ms. In today’s disk technology, the increase in storage capacity of a disk drive has outpaced the increase in performance. Therefore, the performance capacity must be the standard to use when planning an Oracle database’s storage configuration, not disk storage capacity.




The number of disks that should be used is determined first by the I/O requirements then by capacity. This is especially true for datafiles and tempfiles. EFDs can dramatically reduce the number of disks required to perform the I/O required by the workload. Consult with your EMC sales representative for specific sizing recommendations for your workload.

Reserved LUN pool

There is no benefit in assigning LUNs with a high capacity, such as 25 GB to 30 GB, to the reserved LUN pool. It is better to configure the reserved LUN pool with a higher number of LUNs with less capacity (around 5 GB to 8 GB) than with a lower number of LUNs with higher capacity. Approximately 20 to 25 small LUNs are sufficient for most purposes.

CX4 cache configuration for SnapView snapshot

Recommended cache settings

Poor performance was observed using SnapView snapshot with the default settings. This was in the context of a scale-up OLTP workload using a TPC-C-like benchmark. If you have a similar workload and wish to use SnapView snapshot on a CX4 CLARiiON array, you will experience better performance by setting your cache settings as described in the following table of recommended cache settings.

Cache Setting Value

Low watermark 10 percent

High watermark 30 percent

SP A read cache memory 200 MB

SP B read cache memory 200 MB

Write cache memory 1061 MB

Storage processor failover

High availability

The storage processor (SP) failover capability is a key feature that offers redundancy at the storage processor level, allowing continuous data access. It also helps to build a fault-resilient RAC architecture.




LUN/RAID group layout

LUN/RAID group layout design

A LUN/RAID group configuration consisting of three Fibre Channel shelves with RAID 10 and RAID 1 was tested and found to provide good performance for Oracle RAC 11g databases on Celerra. Two RAID and disk configurations were tested over the FCP protocol. These are described below.

RAID group layout: 3 FC shelf RAID 5

The RAID group layout for three-FC shelf RAID 5/RAID 1 is as follows.




RAID group layout: 3 FC shelf RAID 10

The RAID group layout for three-FC shelf RAID 10/RAID 1 is as follows.




Storage design layout

ASM diskgroup guidelines

Automatic Storage Management (ASM) is used to store the database objects requiring high performance. The Oracle Cluster Registry file and the voting disk must be stored on shared storage that is not on the ASM file system; therefore, NFS is used to store these files. In addition, files not requiring high performance are stored on NFS. These NFS file systems are in turn stored on low-cost SATA II drives. This lowers the cost in terms of storage while improving manageability. The following table contains a detailed description of all the database objects and where they should be stored.

File system/ mount point

File system type

LUNs stored on Contents

+DATA ASM LUNs 5 through 10 Oracle datafiles

+LOG1 and +LOG2

ASM LUNs 1 through 4 Online redo logs and control file (mirrored copies)

/u02 NFS LUN 0 Oracle Cluster Registry file and voting disk

/u03 NFS LUN 9 Flashback recovery area (all backups stored here)

/u04 NFS LUN 10 Archived log dump destination

ASM diskgroup design best practice

A diskgroup should consist entirely of LUNs that are all of the same RAID type and that consist of the same number and type of component spindles. EMC does not recommend mixing any of the following within a single ASM diskgroup: • RAID levels • Disk types • Disk rotational speeds

PowerPath/ ASM workaround

The validation was performed using Oracle Enterprise Linux 5.1. We observed the following issue while creating ASM disks:

[root@mteoradb55 ~]# service oracleasm createdisk LUN1 /dev/emcp




Marking disk "/dev/emcpowera1" as an ASM disk: asmtool: Device "/dev/emcpowera1" is not a partition

[FAILED] We used the following workaround to create the ASM disks:

[root@mteoradb55 ~]# /usr/sbin/asmtool -C -l /dev/oracleasm -n LUN1 \

> -s /dev/emcpowera1 -a force=yes

asmtool: Device "/dev/emcpowera1" is not a partition

asmtool: Continuing anyway

[root@mteoradb55 ~]# service oracleasm scandisks

Scanning system for ASM disks: [ OK ]

[root@mteoradb55 ~]# service oracleasm listdisks

LUN1

[root@mteoradb55 ~]# Note PowerPath 5.1 was required for this version of Linux, but this version was not yet GA at the time the validation was started. Therefore, a GA release candidate was used for these tests.

Chapter 3: File System




Overview

Contents This chapter contains the following topic:

Topic See Page

File system layout 31




File system layout

ASM diskgroup guidelines

Automatic Storage Management (ASM) is used to store the database objects requiring high performance. This does not work for the Oracle Cluster Registry file and the voting disk. These two files must be stored on shared storage that is not on the ASM file system. Therefore, NFS is used to store these files. In addition, files not requiring high performance are stored on NFS. These NFS file systems are in turn stored on low-cost SATA II drives. This drives down the cost in terms of storage, while improving manageability.

File system layout

The following table contains a detailed description of all the database objects and where they are stored.

File system/ mount point

File system type

LUNs stored on Contents

/u02 NFS LUN 0 Oracle Cluster Registry file and voting disk

+DATA ASM LUNs 5 through 10 Oracle datafiles

+LOG1 and +LOG2

ASM LUNs 1 through 4 Online redo logs and control file (mirrored copies)

/u03 NFS LUN 9 Flashback recovery area (all backups stored here)

/u04 NFS LUN 10 Archived log dump destination

Chapter 4: Application Design




Overview


Topic See Page

Considerations 33

Application design layout 34

Memory configuration for Oracle 11g 35

HugePages 37




Considerations

Heartbeat mechanisms

The synchronization services component (CSS) of Oracle Clusterware maintains two heartbeat mechanisms: • The disk heartbeat to the voting disk • The network heartbeat across the RAC interconnects that establishes and confirms

valid node membership in the cluster Both of these heartbeat mechanisms have an associated time-out value. For more information on Oracle Clusterware MissCount and DiskTimeout parameters see MetaLink Note 2994430.1UU. EMC recommends setting the disk heartbeat parameter disktimeout to 160 seconds. You should leave the network heartbeat parameter misscount at the default of 60 seconds. Rationale These settings will ensure that the RAC nodes do not evict when the active Data Mover fails over to its partner. The command to configure this option is: $ORA_CRS_HOME/bin/crsctl set css disktimeout 160

https://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=294430.1�




Application design layout

Oracle Cluster Ready Services

Oracle Cluster Ready Services (CRS) are enabled on each of the Oracle RAC 11g/10g servers. The servers operate in active/active mode to provide local protection against a server failure and to provide load balancing. CRS required files (including the voting disk and the OCR file) can reside on NFS volumes provided that the required mount-point parameters are used. For more information on the mount-point parameters required for the Oracle Clusterware files, see Chapter 6: Installation and Configuration > Task 4: Configure NFS client options.

NFS client Each Oracle RAC 11g/10g server, which is hosted in the operating system (OS),

uses the KNFS protocol to connect to the Celerra storage array. KNFS runs over TCP/IP.

Oracle binary files

The Oracle RAC 11g/10g binary files, including the Oracle CRS, are all installed on the database servers' local disks.

Stored on Celerra

Datafiles, tempfiles, online redo logfiles, and controlfiles reside on the FCP file system. Flashback recovery area, disk-based backups, archive logs, and CRS files reside on Celerra NFS file systems. These file systems are designed (in terms of the RAID level and number of disks used) to be appropriate for each type of file. A separate clustered file system is not required for the CRS files. The following table lists each file or activity type and indicates where it resides.

File or activity type Location

Database binary files Database servers local disk (or vmdk file for virtualized servers)

Datafiles, tempfiles +DATA

Online redo log files Mirrored across +LOG1 and +LOG2

Archived log files /archfs

Flashback recovery area /flashfs

Control files Mirrored across +LOG1 and +LOG2

CRS, OCR, and voting disk files /datafs




Memory configuration for Oracle 11g

Memory configuration and performance

Memory configuration in Oracle 11g is one of the most challenging aspects of configuring the database server. If the memory is not configured the performance of the database server will be very poor. If memory is configured incorrectly: • The database server will be unstable • The database may not open at all; and if it does open, you may experience errors

due to lack of shared pool space In an OLTP context, the size of the shared pool is frequently the limitation on performance of the database.

Automatic Memory Management

A new feature called “Automatic Memory Management” was introduced in Oracle 11g 64 bit (Release 1). The purpose of Automatic Memory Management is to simplify the memory configuration process for Oracle 11g. For example, in Oracle 10g, the user is required to set two parameters, SGA_TARGET and PGA_AGGREGATE_TARGET, so that Oracle can manage other memory-related configurations such as buffer cache and shared pool. When using Oracle 11g-style Automatic Memory Management, the user does not set these SGA and PGA parameters. Instead, the following parameters are set: • MEMORY_TARGET • MEMORY_MAX_TARGET Once these parameters are set, Oracle 11g can, in theory, handle all memory management issues, including both SGA and PGA memory. However, the Automatic Memory Management model in Oracle 11g 64 bit (Release 1) requires configuration of shared memory as a file system mounted under /dev/shm. This adds an additional management burden to the DBA/system administrator.

Effects of Automatic Memory Management on performance

Decreased database performance We observed a significant decrease in performance when we enabled the Oracle 11g Automatic Memory Management feature. Linux HugePages are not supported Linux HugePages are not supported when the Automatic Memory Management feature is implemented. When Automatic Memory Management is enabled, the entire SGA memory should fit under /dev/shm and, as a result, HugePages are not used. On both Oracle 11g and Oracle 10g, tuning HugePages increases the performance of the database significantly. It is EMC’s opinion that the performance improvements of HugePages, plus the lack of a requirement for a /dev/shm file system, make the Oracle 11g automatic memory model a poor trade-off.




EMC recommendations To achieve optimal performance on Oracle 11g, EMC recommends the following: • Disable the Automatic Memory Management feature • Use the 10g style of memory management on Oracle 11g The memory management configuration procedure is described in the previous section. This provides optimal performance and manageability per our testing.




HugePages

HugePages The Linux 2.6 kernel includes a feature called HugePages. This feature allows you to

specify the number of physically contiguous large memory pages that will be allocated and pinned in RAM for shared memory segments like the Oracle System Global Area (SGA). The pre-allocated memory pages can only be used for shared memory and must be large enough to accommodate the entire SGA. HugePages can create a very significant performance improvement for Oracle RAC 11g/10g database servers. The performance payoff for enabling HugePages is significant. Warning HugePages must be tuned carefully and set correctly. Unused HugePages can only be used for shared memory allocations - even if the system runs out of memory and starts swapping. Incorrectly configured HugePages settings may result in poor performance and may even make the machine unusable.

HugePages parameters

The HugePages parameters are stored in /etc/sysctl.conf. You can change the value of HugePages parameters by editing the systctl.conf file and rebooting the instance. The following table describes the HugePages parameters:

Parameter Description

HugePages_Total Total number of HugePages that are allocated for shared memory segments (This is a tunable value. You must determine how to set this value.)

HugePages_Free Number of HugePages that are not being used

Hugepagesize Size of each Huge Page Optimum values for HugePages parameters The amount of memory allocated to HugePages must be large enough to accommodate the entire SGA: HugePages_Total x Hugepagessize = Amount of memory allocated to HugePages To avoid wasting memory resources, the value of HugePages_Free should be zero. Note The value of vm.nr_hugepages should be set to a value that is at least equal to kernel.shmmax/2048. When the database is started, the HugePages_Free should show a value close to zero to reflect that memory is tuned. For more information on tuning HugePages, see Chapter 6: Installation and Configuration > Task 7: Tune HugePages.

Chapter 5: Network Design




Overview


Topic See Page

Concepts 39

Best practices 40

SAN network layout 41

IP network layout 41

Virtual LANs 41

Jumbo frames 42

Ethernet trunking and link aggregation 43

Public and private networks 45

Oracle RAC 11g/10g server network architecture 45




Concepts

Jumbo frames Maximum Transfer Unit (MTU) sizes of greater than 1,500 bytes are referred to as

jumbo frames. Jumbo frames require Gigabit Ethernet across the entire network infrastructure – server, switches, and database servers.

VLAN Virtual local area networks (VLANs) logically group devices that are on different

network segments or sub-networks.

Trunking TCP/IP provides the ability to establish redundant paths for sending I/O from one

networked computer to another networked computer. This approach uses the link aggregation protocol, commonly referred to as trunking. Redundant paths facilitate high availability and load balancing for the networked connection.

Trunking device

A trunking device is a virtual device created using two or more network devices to achieve higher performance with load-balancing capability, and high availability with failover capability. With Ethernet trunking/link aggregation, packets traveling through the virtual device are distributed among the underlying devices to achieve higher aggregated bandwidth, based on the source MAC address.




Best practices

Gigabit Ethernet

EMC recommends that you use Gigabit Ethernet for the RAC interconnects if RAC is used. If 10 GbE is available, that is even better.

Jumbo frames and the RAC interconnect

For Oracle RAC 11g/10g installations, jumbo frames are recommended for the private RAC interconnect. This boosts the throughput as well as possibly lowering the CPU utilization due to the software overhead of the bonding devices. Jumbo frames increase the device MTU size to a larger value (typically 9,000 bytes).

VLANs EMC recommends that you use VLANs to segment different types of traffic to

specific subnets. This provides better throughput, manageability, application separation, high availability, and security.




SAN network layout

SAN network layout for validated scenario

The SAN network layout is configured as follows: • Two QLogic FC switches are used for the test bed. • Two connections from each database servers are connected to the QLogic

switches. • One FC port from SPA and SPB are each connected to the two FC switches.

Zoning Each FC port from the database servers are zoned to both SP ports.

IP network layout

IP network design for validated scenario

The IP network layout is configured as follows: • TCP/IP and NFS provide network connectivity. • Client virtual machines run on a VMware ESX server. They are connected to a

client network. • Client, RAC interconnect, and redundant TCP/IP storage networks consist of

dedicated network switches and virtual local area networks (VLANs). • The RAC interconnect and storage networks consist of trunked IP connections to

balance and distribute network I/O. Jumbo frames are enabled on these networks. • The Oracle RAC 11g or 10g servers are connected to the client, RAC interconnect,

WAN, and production storage networks.

Virtual LANs

Virtual LANs This solution uses three VLANs to segregate network traffic of different types. This

improves throughput, manageability, application separation, high availability, and security. The following table describes the database server network port setup:

VLAN ID Description CRS setting

1 Client network Public

2 RAC interconnect Private

3 Storage None (not used)

Client VLAN The client VLAN supports connectivity between the physically booted Oracle RAC

11g/10g servers, the virtualized Oracle Database 11g/10g, and the client




workstations. The client VLAN also supports connectivity between the Celerra and the client workstations to provide network file services to the clients. Control and management of these devices are also provided through the client network.

RAC interconnect VLAN

The RAC interconnect VLAN supports connectivity between the Oracle RAC 11g/10g servers for network I/O required by Oracle CRS. Three network interface cards (NICs) are configured on each Oracle RAC 10g server to the RAC interconnect network. Link aggregation is configured on the servers to provide load balancing and port failover between the two ports for this network.

Redundant switches

In addition to VLANs, separate redundant storage switches are used. The RAC interconnect connections are also on a dedicated switch. For real-world solution builds, it is recommended that these switches support Gigabit Ethernet (GbE) connections, jumbo frames, and port channeling.

Jumbo frames

Overview Jumbo frames are configured for the following layers:

• Celerra Data Mover • Oracle RAC 11g/10g servers • Switch Note Configuration steps for the switch are not covered here, as that is vendor-specific. Check your switch documentation for details.

Linux servers To configure jumbo frames on a Linux server, execute the following command:

ifconfig eth0 mtu 9000 Alternatively, place the following statement in the network scripts in /etc/sysconfig/network-scripts: MTU=9000

RAC interconnect

Jumbo frames should be configured for the storage and RAC interconnect networks of this solution to boost the throughput, as well as possibly lowering the CPU utilization due to the software overhead of the bonding devices. Jumbo frames increase the device MTU size to a larger value (typically 9,000 bytes). Typical Oracle database environments transfer data in 8 KB and 32 KB block sizes, which require multiple 1,500 frames per database I/O, while using an MTU size of 1,500. Using jumbo frames, the number of frames needed for every large I/O request can be reduced, thus the host CPU needed to generate a large number of interrupts for each application I/O is reduced. The benefit of jumbo frames is primarily a complex function of the workload I/O sizes, network utilization, and Oracle database




server CPU utilization, and so is not easy to predict. For information on using jumbo frames with the RAC Interconnect, see MetaLink Note 300388.1.

Verifying that jumbo frames are enabled

To test whether jumbo frames are enabled, use the following command: ping –M do –s 8192 <target> Where: target is the interface to be tested Jumbo frames must be enabled on all layers of the network for this command to succeed.

Ethernet trunking and link aggregation

Trunking and link aggregation

Two NICs on each Oracle RAC 11g/10g server are used in the NFS connection, referred to previously as the storage network. The RAC interconnect network is trunked in a similar manner using three NICs. EMC recommends that you configure an Ethernet trunking interface with two Gigabit Ethernet ports to the same switch.

Enabling trunking on a Linux database server

On the database servers, network redundancy is achieved by using Linux kernel bonding. This is accomplished using the scripts contained in /etc/sysconfig/network-scripts. A typical bonded connection is as follows: DEVICE=bond0

ONBOOT=yes

BOOTPROTO=none

USERCTL=no

IPADDR=10.6.25.188

NETMASK=255.255.255.0

MTU=9000 This device (bond0) consists of two Ethernet ports, whose scripts are similar to the following: DEVICE=eth1

MASTER=bond0

SLAVE=yes

ONBOOT=yes

BOOTPROTO=none

HWADDR=00:04:23:B9:66:F3 The result is that the Ethernet ports that show their master as bond0 are joined to the bonded connection. Modify the /etc/modprobe.conf file.

https://metalink.oracle.com/metalink/plsql/f?p=130:14:946500529267365575::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,300388.1,1,1,1,helvetica�




The following is an example of the lines that must be added: options bonding max_bonds=2 mode=4

alias bond0 bonding

alias bond1 bonding Either reboot the Linux server or down and up the interfaces to enable the trunk.




Public and private networks

Public and private networks

Each node should have: • One static IP address for the public network • One static IP address for the private cluster interconnect The private interconnect should only be used by Oracle to transfer cluster manager and cache fusion-related data. Although it is possible to use the public network for the RAC interconnect, this is not recommended as it may cause degraded database performance (reducing the amount of bandwidth for cache fusion and cluster manager traffic).

Configuring virtual IP addresses

The virtual IP addresses must be defined in either the /etc/hosts file or DNS for all RAC nodes and client nodes. The public virtual IP addresses will be configured automatically by Oracle when the Oracle Universal Installer is run, which starts Oracle's Virtual Internet Protocol Configuration Assistant (vipca). All virtual IP addresses will be activated when the following command is run: srvctl start nodeapps -n <node_name> Where: node_name is the hostname/IP address that will be configured in the client's tnsnames.ora file.

Oracle RAC 11g/10g server network architecture

Oracle RAC 11g/10g server network interfaces - NFS

The following table lists each interface and describes its use for the Oracle 11g/10g NFS configuration.

Interface port ID Description

eth0 Client network

eth1 Unused

eth2 Unused

eth3 Unused

eth4 Storage network (trunked)

eth5 Storage network (trunked)

eth6 Unused




eth7 RAC interconnect (trunked)



Oracle RAC 11g/10g server network interfaces - FCP

The following tables list each server and describe its use for the Oracle 11g/10g FCP configuration. There are two dual FC port host bus adapters on each of the database servers. Two FC ports from each of these database servers are connected to one of the QLogic FC switches. The other two FC ports are connected to a different switch for high availability. One port from SPA and one port from SPB are connected to each of the two FC switches.

Database Server Description

HBA Port 0 QLogic FC Switch-1




CLARiiON Description

SPA Port 0 QLogic FC Switch-1

SPB Port 1 QLogic FC Switch-1

SPA Port 1 QLogic FC Switch-2

SPB Port 0 QLogic FC Switch-2

Chapter 6: Installation and Configuration




Overview

Introduction This chapter provides procedures and guidelines for installing and configuring the

components that make up the validated solution scenario.

Scope The installation and configuration instructions presented in this chapter apply to the

specific revision levels of components used during the development of this solution. Before attempting to implement any real-world solution based on this validated scenario, gather the appropriate installation and configuration documentation for the revision levels of the hardware and software components as planned in the solution. Version-specific release notes are especially important.

Contents This chapter contains the following tasks:

Topic See Page

Task 1: Build the network infrastructure 48

Task 2: Set up and configure ASM for CLARiiON 48

Task 3: Set up and configure database servers 49

Task 4: Configure NFS client options 50

Task 5: Install Oracle Database 11g/10g 51

Task 6: Configure database server memory options 51

Task 7: Tune HugePages 54

Task 8: Set database initialization parameters 56

Task 9: Configure Oracle Database control files and logfiles 58

Task 10: Enable passwordless authentication using SSH 59

Task 11: Set up and configure CLARiiON storage for Replication Manager and SnapView

64

Task 12: Install and configure EMC RecoverPoint 67

Task 13: Set up the virtualized utility servers 70

Task 14: Configure and connect EMC RecoverPoint appliances (RPAs)

71

Task 15: Install and configure EMC MirrorView/A 71

Task 16: Install and configure EMC CLARiiON (CX) splitters 71




Task 1: Build the network infrastructure

Network infrastructure

For details on building a network infrastructure, see Chapter 5: Network Design > IP network layout > IP network design for validated scenario.

Task 2: Set up and configure ASM for CLARiiON

Configure ASM and manage CLARiiON

For details on configuring ASM and managing the CLARiiON, follow the steps in the table below.

Step Action

1 Find the operating system (OS) version.

[root@mteoradb55 ~]# cat /etc/redhat-release

Enterprise Linux Enterprise Linux Server release 5.1 (Carthage)

[root@mteoradb55 ~]#

2 Check the PowerPath installation.

[root@mteoradb55 ~]# rpm -qa EMC*

EMCpower.LINUX-5.1.0-194


3 Check the ASM rpms applied on the OS.

[root@mteoradb55 ~]# rpm -qa | grep oracleasm

oracleasm-support-2.0.4-1.el5

oracleasmlib-2.0.3-1.el5

oracleasm-2.6.18-53.el5-2.0.4-1.el5


4 Configure the ASM.

[root@mteoradb55 ~]# /etc/init.d/oracleasm configure

5 Check the status of ASM.

[root@mteoradb55 ~]# /etc/init.d/oracleasm status

6 Create the ASM disk.

[root@mteoradb55 ~]# /etc/init.d/oracleasm createdisk LUN1 /dev/emcpowerab

7 Scan the ASM disk.

[root@mteoradb55 ~]# /etc/init.d/oracleasm scandisks

8 List the ASM disks.

[root@mteoradb55 ~]# /etc/init.d/oracleasm listdisks




Task 3: Set up and configure database servers

Check BIOS version

Dell PowerEdge 2900 servers were used in our testing. These servers were preconfigured with the A06 BIOS. Upgrading the BIOS to the latest version (2.2.6 as of the time of this publication) resolved a range of issues, including hanging reboot problems and networking issues. Regardless of the server vendor and architecture, you should monitor the BIOS version shipped with the system and determine if it is the latest production version supported by the vendor. If it is not the latest production version supported by the vendor, then flashing the BIOS is recommended.

Disable Hyper-Threading

Intel Hyper-Threading Technology allows multi-threaded operating systems to view a single physical processor as if it were two logical processors. A processor that incorporates this technology shares CPU resources among multiple threads. In theory, this enables faster enterprise-server response times and provides additional CPU processing power to handle larger workloads. As a result, server performance will supposedly improve. In EMC’s testing, however, performance with Hyper-Threading was poorer than performance without it. For this reason, EMC recommends disabling Hyper-Threading. There are two ways to disable Hyper-Threading: in the kernel or through the BIOS. Intel recommends disabling Hyper-Threading in the BIOS because it is cleaner than doing so in the kernel. Refer to your server vendor’s documentation for instructions.




Task 4: Configure NFS client options

NFS client options

For optimal reliability and performance, EMC recommends the NFS client options listed in the table below. The mount options are listed in the /etc/fstab file.

Option Syntax Recommended Description

Hard mount hard Always The NFS file handles are kept intact when the NFS server does not respond. When the NFS server responds, all the open file handles resume, and do not need to be closed and reopened by restarting the application. This option is required for Data Mover failover to occur transparently without having to restart the Oracle instance.

NFS protocol version

vers= 3 Always Sets the NFS version to be used. Version 3 is recommended.

TCP proto=tcp Always All the NFS and RPC requests will be transferred over a connection-oriented protocol. This is required for reliable network transport.

Background bg Always Enables client attempts to connect in the background if the connection fails.

No interrupt nointr Always This toggle allows or disallows client keyboard interruptions to kill a hung or failed process on a failed hard-mounted file system.

Read size and write size

rsize=32768, wsize=32768

Always Sets the number of bytes NFS uses when reading or writing files from an NFS server. The default value is dependent on the kernel. However, throughput can be improved greatly by setting rsize/wsize= 32768

No auto noauto Only for backup/utility file systems

Disables automatic mounting of the file system on boot-up. This is useful for file systems that are infrequently used (for example, stage file systems).

Timeout timeo=600 Always Sets the time (in tenths of a second) the NFS client waits for the request to complete.




sunrpc.tcp_slot_table_entries

The NFS module called “sunrpc.tcp_slot_table_entries” controls the concurrent I/Os to the storage system. The default value of this parameter is 16. The parameter should be set to the maximum value (128) for enhanced I/O performance. To configure this option, type the following command: [root@mteoraesx2-vm3 ~]# sysctl -w sunrpc.tcp_slot_table_entries=128 sunrpc.tcp_slot_table_entries = 128

Important Before configuring this option, you must make the changes in sysctl.conf, and then run sysctl –w. This reparses the file, and the resulting text is output.

Task 5: Install Oracle Database 11g/10g

Install Oracle Database 11g for Linux

See Oracle’s installation guide: Oracle Database Installation Guide 11g Release 1 (11.1) for Linux

Install Oracle Database 10g for Linux

See Oracle’s installation guide: Oracle Database Client Installation Guide 10g Release 1 (10.1.0.3) for Linux x86-64

Task 6: Configure database server memory options

Database server memory

Refer to your database server documentation to determine the total number of memory slots your database server has, and the number and density of memory modules that you can install. EMC recommends that you configure the system with the maximum amount of memory feasible to meet the scalability and performance needs. Compared to the cost of the remaining components in an Oracle database server configuration, the cost of memory is minor. Configuring an Oracle database server with the maximum amount of memory is entirely appropriate.

Shared memory Oracle uses shared memory segments for the Shared Global Area (SGA), which is

an area of memory that is shared by Oracle processes. The size of the SGA has a significant impact on the database performance, and there is a direct correlation between SGA size and disk I/O. EMC’s Oracle RAC 11g/10g testing was done with servers using 20 GB of SGA.

http://www.oracle.com/pls/db111/to_toc?pathname=install.111/b32002/toc.htm�

http://download.oracle.com/docs/html/B14400_01/title.htm�




Memory configuration files

The following table describes the files that must be configured for memory management:

File Created by Function

/etc/sysctl.conf Linux installer Contains the shared memory parameters for the Linux operating system. This file must be configured in order for Oracle to create the SGA with shared memory.

/etc/security/limits.conf

Linux installer Contains the limits imposed by Linux on users’ use of resources. This file must be configured correctly in order for Oracle to use shared memory for the SGA.

Oracle parameter file

Oracle installer, dbca, or DBA who creates the database

Contains the parameters used by Oracle to start an instance. This file must contain the correct parameters in order for Oracle to start an instance using shared memory.

Configuring /etc/sysctl.conf

Configure the etc/sysctl.conf file as follows:

# Oracle parameters

kernel.shmall = 33554432

kernel.shmmax = 137438953472

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 6553600

net.ipv4.ip_local_port_range = 1024 65000

net.core.rmem_default = 4194304

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 262144

vm.nr_hugepages = 51203

sunrpc.tcp_slot_table_entries = 128 Recommended parameter values The following table describes recommended values for kernel parameters.




Kernel parameter

Parameter function Recommended value

kernel.shmmax Defines the maximum size in bytes of a single shared memory segment that a Linux process can allocate in its virtual address space. Since the SGA is comprised of shared memory, SHMMAX can potentially limit the size of the SGA.

33554432 (Slightly larger than the SGA size)

kernel.shmini Sets the system-wide maximum number of shared memory segments.

4096

kernel.shmall The value should be at least ceil(shmmax/PAGE_SIZE). The PAGE_SIZE on our Linux systems was 4096.

137438953472

Configuring /etc/security/limits.conf

The section of the /etc/security/limits.conf file relevant to Oracle should be configured as follows: # Oracle parameters

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536

oracle soft memlock 8000000

oracle hard memlock 8000000 Important Ensure that the memlock parameter has been configured. This is required for the shared memory file system. This is not covered in the Oracle Database 11g Installation Guide, so be sure to set this parameter. If you do not set the memlock parameter, your database will behave uncharacteristically.




Task 7: Tune HugePages

Tuning HugePages

The following table describes how to tune HugePages parameters to ensure optimum performance.

Step Action

1 Ensure that the machine you are using has adequate memory. For example, our test system had 24 GB of RAM and a 20 GB SGA.

2 Set the HugePages parameters in /etc/sysctl.conf to a size into which the SGA will fit comfortably. For example, to create a HugePages pool of 21 GB, which would be large enough to accommodate the SGA, set the following parameter values: HugePages_Total: 11241 Hugepagesize: 2048 kB

3 Reboot the instance.

4 Check the values of the HugePages parameters by typing the following command: [root@mteoradb51 ~]# grep Huge /proc/meminfo

On our test system, this command produced the following output: HugePages_Total: 11241

HugePages_Free: 1000

Hugepagesize: 2048 KB

5 If the value of HugePages_Free is equal to zero, the tuning is complete: If the value of HugePages_Free is greater than zero: a) Subtract the value of HugePages_Free from HugePages_Total.

Make note of the answer. b) Open /etc/sysctl.conf and change the value of HugePages_Total to

the answer you calculated in step a). c) Repeat steps 3, 4, and 5.

Tuning HugePages on RHEL 5/OEL 5

On Red Hat Enterprise Linux 5 and on Oracle Enterprise Linux 5 systems, HugePages cannot be configured using the steps mentioned above. We used a shell script called “hugepage_settings.sh” to configure HugePages on these systems. This script is available on Oracle MetaLink Note 401749.1. The hugepage_settings.sh script configures HugePages as follows: HugePages_Total: 12804

HugePages_Free: 2244

HugePages_Rsvd: 2240

Hugepagesize: 2048 kB

https://metalink2.oracle.com/metalink/plsql/f?p=130:14:7593786276690194066::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,401749.1,1,1,1,helvetica�




More information about HugePages

For more information on enabling and tuning HugePages, refer to: • Oracle MetaLink Note 361323.1 • Tuning and Optimizing Red Hat Enterprise Linux for Oracle 9i and 10g Databases

https://metalink.oracle.com/CSP/main/article?cmd=show&type=NOT&id=361323.1�

http://www.puschitz.com/TuningLinuxForOracle.shtml�




Task 8: Set database initialization parameters

Overview This section describes the initialization parameters that should be set in order to

configure the Oracle instance for optimal performance on the CLARiiON CX4 series. These parameters are stored in the spfile or init.ora file for the Oracle instance.

Database block size

Parameter Database block size

Syntax DB_BLOCK_SIZE=n

Description For best database performance, DB_BLOCK_SIZE should be a multiple of the OS block size. For example, if the Linux page size is 4096, DB_BLOCK_SIZE =4096 *n.

Direct I/O Parameter Direct I/O

Syntax FILESYSTEM_IO_OPTIONS=setall

Description This setting enables direct I/O and async I/O. Direct I/O is a feature available in modern file systems that delivers data directly to the application without caching in the file system buffer cache. Direct I/O preserves file system semantics and reduces the CPU overhead by decreasing the kernel code path execution. I/O requests are directly passed to network stack, bypassing some code layers. Direct I/O is a very beneficial feature to Oracle’s log writer, both in terms of throughput and latency. Async I/O is beneficial for datafile I/O.

Multiple database writer processes

Parameter Multiple database writer processes

Syntax DB_WRITER_PROCESSES=2*n

Description The recommended value for db_writer_processes is that it at least match the number of CPUs. During testing, we observed very good performance by just setting db_writer_processes to 1.

Multi Block Read Count

Parameter Multi Block Read Count

Syntax DB_FILE_MULTIBLOCK_READ_COUNT= n

Description DB_FILE_MULTIBLOCK_READ_COUNT determines the maximum number of database blocks read in one




I/O during a full table scan. The number of database bytes read is calculated by multiplying the DB_BLOCK_SIZE by the DB_FILE_MULTIBLOCK_READ_COUNT. The setting of this parameter can reduce the number of I/O calls required for a full table scan, thus improving performance. Increasing this value may improve performance for databases that perform many full table scans, but degrade performance for OLTP databases where full table scans are seldom (if ever) performed. Setting this value to a multiple of the NFS READ/WRITE size specified in the mount limits the amount of fragmentation that occurs in the I/O subsystem. This parameter is specified in DB Blocks and NFS settings are in bytes - adjust as required. EMC recommends that DB_FILE_MULTIBLOCK_READ_COUNT be set to between 1 and 4 for an OLTP database and to between 16 and 32 for DSS.

Disk Async I/O Parameter Disk Async I/O

Syntax DISK_ASYNCH_IO=true

Description RHEL 4 update 3 and later support async I/O with direct I/O on NFS. Async I/O is now recommended on all the storage protocols.

Use Indirect Memory Buffers

Parameter Use Indirect Memory Buffers

Syntax USE_INDIRECT_DATA_BUFFERS=true

Description Required to support the use of the /dev/shm in-memory file system for storing the SGA shared memory structures.




Task 9: Configure Oracle Database control files and logfiles

Control files EMC recommends the following:

• When you create the control file, allow for growth by setting MAXINSTANCES,

MAXDATAFILES, MAXLOGFILES, and MAXLOGMEMBERS to high values. Your database should have a minimum of two control files located on separate physical ASM diskgroups. One way to multiplex your control files is to store a control file copy on every diskgroup that stores members of the redo log groups.

Online and archived redo log files

EMC recommends that you: • Run a mission-critical, production database in ARCHIVELOG mode. • Multiplex your redo log files for these databases. Loss of online redo log files could

result in a database recovery failure. The best practice to multiplex your online redo log files is to place members of a redo log group on different ASM diskgroups.

To understand how redo log and archive log files can be placed, refer to the Reference Architecture diagram.




Task 10: Enable passwordless authentication using SSH

Overview The use of passwordless authentication using ssh is a fundamental concept to make

successful use of Oracle RAC 10g or 11g with Celerra.

SSH files SSH passwordless authentication relies on the three files described in the following

table.

File Created by Purpose

~/.ssh/id_dsa.pub ssh-keygen Contains the host’s dsa key for ssh authentication (functions as the proxy for a password)

~/.ssh/authorized_keys ssh Contains the dsa keys of hosts that are authorized to log in to this server without issuing a password

~/.ssh/known_hosts ssh Contains the dsa key and hostname of all hosts that are allowed to log in to this server using ssh

id_dsa.pub The most important ssh file is id_dsa.pub.

Important If the id_dsa.pub file is re-created after you have established a passwordless authentication for a host onto another host, the passwordless authentication will cease to work. Therefore, do not accept the option to overwrite id_dsa.pub if ssh-keygen is run and it discovers that id_dsa.pub already exists.

Enabling authentication: Single user/single host

The following table describes how to enable passwordless authentication using ssh for a single user on a single host:

Step Action

1 Create the dsa_id.pub file using ssh-keygen.

2 Copy the key for the host for which authorization is being given to the authorized_keys file of the host that allows the login.

3 Complete a login so that ssh knows about the host that is logging in. That is, record the host’s key and hostname in the known_hosts file.




Enabling authentication: Single user/multiple hosts

Prerequisites To enable authentication for a user on multiple hosts, you must first enable authentication for the user on a single host: Chapter 6: Installation and Configuration > Task 10: Enable passwordless authentication using SSH > Enabling authentication: Single user/single host Procedure summary After you have enabled authentication for a user on a single host, you can then enable authentication for the user on multiple hosts by copying the authorized_keys and known_hosts files to the other hosts. This is a very common task when setting up Oracle RAC 11g//10g prior to installation of Oracle Clusterware. It is possible to automate this task by using the ssh_multi_handler.bash script. ssh_multi_handler.bash #!/bin/bash

#-----------------------------------------------------------#

# Script: ssh_multi_handler.bash #

# Purpose: Handles creation of authorized_keys #

#-----------------------------------------------------------#

ALL_HOSTS="rtpsol347 rtpsol348 rtpsol349 rtpsol350"

THE_USER=root

mv -f ~/.ssh/authorized_keys ~/.ssh/authorized_keys.bak

mv -f ~/.ssh/known_hosts ~/.ssh/known_hosts.bak

for i in ${ALL_HOSTS}

do

ssh ${THE_USER}@${i} "ssh-keygen -t dsa"

ssh ${THE_USER}@${i} "cat ~/.ssh/id_dsa.pub" \

>> ~/.ssh/authorized_keys

ssh ${THE_USER}@${i} date

done

for i in $ALL_HOSTS

do

scp ~/.ssh/authorized_keys ~/.ssh/known_hosts \

${THE_USER}@${i}:~/.ssh/

done




for i in ${ALL_HOSTS}

do

for j in ${ALL_HOSTS}

do

ssh ${THE_USER}@${i} "ssh ${THE_USER}@${j} date"

done

done

mv -f ~/.ssh/authorized_keys.bak ~/.ssh/authorized_keys

mv -f ~/.ssh/known_hosts.bak ~/.ssh/known_hosts

exit How to use ssh_multi_handler.bash At the end of the process described below, all of the equivalent users on the set of hosts will be able to log in to all of the other hosts without issuing a password.

Step Action

1 Copy and paste the text from ssh_multi_handler.bash into a new file on the Linux server.

2 Edit the variable definitions at the top of the script.

3 Use chmod on the script to allow it to be executed.

4 Run the script. Output on our systems On our systems with the settings noted previously, this script produced the following effect: ssh multi-host output [root@rtpsol347 ~]# ./ssh_multi_handler.bash

Enter file in which to save the key (/root/.ssh/id_dsa):

Generating public/private dsa key pair.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_dsa.

Your public key has been saved in /root/.ssh/id_dsa.pub.

The key fingerprint is:

f8:21:61:55:55:92:15:ed:0a:62:89:c5:ed:93:5f:27 [email protected]

root@rtpsol347's password:

Tue Aug 8 22:21:31 EDT 2006





...(additional similar output not shown)

authorized_keys 100% 5008 4.9KB/s 00:00

known_hosts 100% 2189 2.1KB/s 00:00


...<repeated 3 times>

Tue Aug 8 22:22:05 EDT 2006

...<repeated 15 times>

[root@rtpsol347 ~]# The 16 date outputs, without any requests for passwords, indicate that the passwordless authentication files on all root users among these four hosts have been successfully created.

Enabling authentication: Single host/different user

Another common task is to set up passwordless authentication across two users between two hosts. For example, enable the Oracle user on the database server to run commands as the root or nasadmin user on the Celerra Control Station. You can set this up by using the ssh_single_handler.bash script. This script creates passwordless authentication from the presently logged in user to the root user on the Celerra Control Station. ssh_single_handler.bash

#!/bin/bash

#-----------------------------------------------------------#

# Script: ssh_single_handler.bash #

# Purpose: Handles creation of authorized_keys #

#-----------------------------------------------------------#

THE_USER=root

THE_HOST=rtpsol33

ssh-keygen -t dsa

KEY=`cat ~/.ssh/id_dsa.pub`

ssh ${THE_USER}@${THE_HOST} "echo ${KEY} >> \

~/.ssh/authorized_keys"

ssh ${THE_USER}@${THE_HOST} date

exit




Output on our systems On our systems with the settings noted previously, ssh_single_handler.bash produced the following effect: ssh single host output [oracle@rtpsol347 scripts]$ ./ssh_single_handler.bash

Generating public/private dsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_dsa.

Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.

The key fingerprint is:

09:13:4d:7d:20:0c:9a:c4:4e:35:c9:c9:11:9e:30:31 [email protected]

Wed Aug 9 09:40:01 EDT 2006

[oracle@rtpsol347 scripts]$

The date output without a password request indicates that the passwordless authentication files have been created.




Task 11: Set up and configure CLARiiON storage for Replication Manager and SnapView

Setup and configuration of CLARiiON storage

Carry out the steps in the following table to configure CLARiiON storage to use Replication Manager (RM) and SnapView to create clones for backup and recovery and test/dev.

Note To enable the CLARiiON cloning feature, two LUNs with a minimum capacity of 1 GB (for CX4 series) must be designated as clone private LUNs. These LUNs will be used during cloning.

Step Action

1 Create a storage group: a. In Navisphere Manager, click the Storage tab, right-click Storage Groups and select Create Storage Group. b. In the Storage Group Name field, enter “EMC Replication Storage”.

The name “EMC Replication Storage” must be used in order for Replication Manager to work correctly. The name is case sensitive.

c. Click Apply, then click OK.

2 Add LUNs to the storage group: a. In Navisphere Manager, under Storage Groups, right-click EMC Replication Storage and choose Select LUNs.

(The Storage Group Properties tab is displayed.) b. In the LUNs tab, select all the LUNs that are used to store the database, then click Apply.

3 Create a separate storage group for clone target LUNs: a. In Navisphere Manager, click the Storage tab, right-click Storage Groups and select Create Storage Group. b. In the Storage Group Name field, enter “Mount Host SG”.

4 Add target LUNs to the Mount Host SG storage group: a. Select Mount Host SG and right-click. b. Click Select LUNs.

(The Storage Group Properties tab is displayed.) c. Select the target LUNs to be used for cloning, then click Apply.

5 Add the host to the clone target storage group. This allows the clone target LUNs to be mounted on the target storage group to bring up the test/dev copy of the database.




Configuring Replication Manager

The procedure for preparing and installing Replication Manager (RM) components is outside the scope of this document. After installing Replication Manager Server and Agents, carry out the steps in the following tables to: • Add a host • Add storage • Create a LUNs pool • Create an application set

Adding a host using RM

The following table lists the steps for adding a host using Replication Manager.

Step Action

1 In the Replication Manager console, select and right-click Hosts, then select New Host.

2 Type a name for the host (database server name), then click OK.

3 Repeat steps 1 and 2 to add all other hosts (database servers) and the mount host server to Replication Manager.

Adding storage using RM

After you have added the hosts and the mount host, you must add the storage. The following table lists the steps for adding storage using Replication Manager.

Step Action

1 In Replication Manager, select and right-click Storage Services, then select Add Storage.

2 To start the Add Storage Wizard, in the Confirm dialog box, click Yes, then click Next to go to the next screen.

3 On the Select Storage Services screen, select the appropriate storage service and double-click it.

4 On the Array Connections screen, enter the login credentials for the storage array, then click OK.

5 When the Discover Storage screen progress bar reaches 100%, click Close.

6 On the Select Target Devices screen, select the snap cache for snapshots and the LUNs, then click Next.

7 To complete the procedure, click Finish.

8 When the Add Storage screen progress bar reaches 100%, click Close. The storage array is now visible in the Storage Services list.




Creating a LUNS pool

A dedicated storage pool for cloning must be created so that the specified LUNs in that pool can be selected as cloning targets. To create a new storage pool, carry out the steps in the following table:

Step Action

1 In Replication Manager, select and right-click Storage Pools, then select New Storage Pool.

2 On the New Pool screen, enter a pool name and a description, then click Add.

3 Select the LUNs that you want to include in the storage pool and click OK.

4 Click Add to add the updated LUNs to the new pool, then click Close.

Creating an application set using RM

After you have added the storage to Replication Manager, you must then create an application set using the Replication Manager console. The following table lists the steps for creating an application set:

Step Action

1 In Replication Manager, select and right-click Application Sets, then select New Application Set.

2 In the Add Application Set wizard, click Next to go to the next screen.

3 On the Application Set Name and Objects screen, click Add Instance to create a database instance.

4 On the Application Credentials screen, enter the Oracle database credentials or the ASM credentials, then click OK. The database instance created.

5 Select the newly created instance and type a name for the application set, then click Next.

6 On the Completing the Application Set wizard screen, click Finish. The application set is created.




Task 12: Install and configure EMC RecoverPoint

Installation When performing the following steps, the RecoverPoint GUI provides many options.

A detailed explanation of those options is available in the EMC RecoverPoint Administrator’s Guide. Refer to the EMC RecoverPoint Installation Guide for full installation instructions.

Configuring RecoverPoint

Configuration requires one storage group per site, which consists of the repository volume, journal volumes, and all the replication volumes. The repository volume and journal volumes will not be accessible from any hosts other than the RPAs. But the RPAs will need to have access to all the volumes that will take part in replication. Similarly, there must be one more storage group consisting of hosts and the replication volumes to which the hosts will have access. The hosts should not have access to repository and journal volumes. The steps for validating the disaster recovery solution using RecoverPoint are described in the following table.

Step Action

1 Zone all the RPA ports with all the CLARiiON array ports. A single zone consisting of all RPA ports and all CLARiiON ports per site should be created.

2 Manually register the RPA initiators as “Initiator Type: RecoverPoint Appliance” with the failover mode set to “4”.

3 Complete the installation of the RPAs. Refer to Chapter 2 of the EMC RecoverPoint Installation Guide for instructions. Start from the step after booting up the RPA with the new software and continue up to, but not including, attaching the splitters. During our testing, the RecoverPoint appliances at both sites were configured as shown below: RPA Setup Details Parameter Site 1: Value Site 2: Value Site Name Source Target Management Default Gateway 10.6.115.1 10.6.115.1 Management Subnet Mask 255.255.255.0 255.255.255.0 WAN default gateway 192.168.4.1 192.168.4.1 WAN subnet mask 255.255.255.0 255.255.255.0 Site Management IP 10.6.115.125 10.6.115.126 Time zone America/New_York America/New_York Primary DNS server 10.6.24.15 10.6.24.15 Secondary DNS server 10.6.29.16 10.6.29.16 Local domain solutions1.rtp.dg.com solutions1.rtp.dg.com




NTP server 10.6.24.62 N/A Number of virtual ports N/A N/A Initiator only mode N/A N/A Number of exposed LUNs N/A N/A Box 1 Box Management IP 10.6.115.50 10.6.115.69 Box WAN IP 192.168.4.150 192.168.4.160 Remote maintenance port N/A N/A Box 2 Box Management IP 10.6.115.51 10.6.115.70 Box WAN IP 192.168.4.151 192.168.4.161 Remote maintenance port N/A N/A

4 Verify the RPA to CX splitter connectivity by logging in to the CLI of the RPA. You can verify the RPA to CX splitter connectivity using the following option:

Run: [3]:Diagnostics > [2]:Fibre Channel diagnostics > [4]:Detect Fibre Channel LUNs

If connectivity has been established, you will see the FC LUNs.

5 To add new splitters to the RPA environment, right-click on the splitter icon in the RecoverPoint GUI. Splitters can then be selected and added. Splitters must be added for both sites.

6 Create a consistency group: a) In the navigation pane of the RecoverPoint management console,

select Consistency Groups. b) In the Consistency Groups pane, click Add new group. A detailed description of the different options that can be specified for a consistency group is available in Chapter 2 of the EMC RecoverPoint Administrator’s Guide.

7 Configure the source and target copies: a) In the navigation pane of the RecoverPoint management console,

select Consistency Groups > Oracle DB Consistency Group. b) In the Oracle DB Consistency Group pane, select the Status tab. c) Click the Add Copy icon. The New Copy dialog box is displayed.

8 In the New Copy dialog box, select the site of the production source and enter the values for General Settings and Advanced Settings. Refer to the EMC RecoverPoint Administrator’s Guide for a detailed explanation of all the available options.




9 Define the replication sets: a) In the navigation pane, select Volumes. b) Click Configuration.

The Volumes Configuration dialog box is displayed. c) To create a new replication set, click Add New Replication Set. d) Add a volume to both the source-side replica and target-side

replica. e) Add at least one journal volume per replication set.

10 Attach volumes to the splitters: a) In the navigation pane, select Splitters.

The Splitters tab with the available splitters is displayed. b) Select a splitter and double-click. c) Click Rescan and then click Attach.

The replication sets that are discovered by the splitter are displayed.

d) Select the replication sets that you want to add to the splitters. Note A volume cannot be replicated unless it is attached to a splitter. Once the replication sets have been added, the Splitter Properties screen will display all the replication sets that are associated with the splitters.

11 Enable a consistency group by selecting its name in the navigation panel and clicking Enable Group. Note The consistency groups should be enabled before starting replication.

12 Start replication from source to target site by clicking Start Transfer.

13 Enable image access. Once the replication is complete, the target image can be accessed by selecting Image Access from the drop-down menu.




Task 13: Set up the virtualized utility servers

Setting up the virtualized utility servers

Virtualized single instance database servers were used as targets for test/dev and disaster recovery solutions. To set up a virtualization configuration, you need to do the steps outlined in the following table:

Step Action

1 Deploy a VMware ESX server.

2 Capture the total physical memory and total number of CPUs that are available on the ESX server.

3 Create four virtual machines (VMs) on the ESX server. For the storage network configuration, see Chapter 8: Virtualization > VMware ESX server > Typical storage network configuration.

4 Distribute the memory and CPUs available equally to each of the VMs.

5 Assign a VMkernel IP (10.6.115.144) to each ESX server so that it can be used to mount NFS storage. For the storage configuration, see Chapter 8: Virtualization > VMware ESX server > Storage configuration.

Note All the VMs need to be located on common storage. This is mandatory for performing VMotion.

6 Configure four additional NICs on the ESX server; dedicate each NIC to a VM. These additional NICs are used to configure the dedicated private network connection to Celerra where the database files reside.

7 Ensure all necessary software, for example, Oracle, is installed and configured.

Note All database objects are stored on an NFS mount.




Task 14: Configure and connect EMC RecoverPoint appliances (RPAs)

Configuring RecoverPoint appliances (RPAs)

Refer to the EMC RecoverPoint Installation Guide, located on Powerlink.com, for full installation and configuration instructions. Access to this document is based on your login credentials. If you do not have access, contact your EMC representative.

Task 15: Install and configure EMC MirrorView/A

Installing MirrorView/A

Refer to the EMC MirrorView Installation Guide, located on Powerlink.com, for full installation instructions. Access to this document is based on your login credentials. If you do not have access, contact your EMC representative.

Task 16: Install and configure EMC CLARiiON (CX) splitters

Installing CX splitters

Refer to the section on installing CLARiiON (CX) splitters in the EMC RecoverPoint Installation Guide, located on Powerlink.com, for full installation instructions. Access to this document is based on your login credentials. If you do not have access, contact your EMC representative.

Chapter 7: Testing and Validation




Overview

Introduction to testing and validation

This chapter provides a detailed summary and description of the tests performed to validate an EMC Proven™ Solution. The goal of the testing was to record the response of the end-to-end solution and component subsystem under reasonable load. The solution was tested under a load that is representative of the market for Oracle RAC 11g/10g on Linux with an EMC Celerra unified storage platform over Fibre Channel Protocol (FCP) and Network File System (NFS).

Objectives The objectives of this testing were to carry out:

• Performance testing of a blended solution using FCP for datafiles, online redo log

files, and controlfiles (all of the files in the Oracle database environment that require high-performance I/O), and NFS for all the other files required to be stored by the Oracle database environment (archived log files, OCR files, flashback recovery area, and database backups).

• Functionality testing of a test/dev solution using a blended configuration, whereby a cloned version of a physically-booted production Oracle RAC 11g/10g database was replicated and then mounted on a VMware virtual machine running a single-instance Oracle Database 11g/10g.

Cycling issues and peak performance

During testing, we observed cycling issues at higher user loads. For the most part, these issues were related to the Quest Benchmark Factory application, which was used to generate the TPC-C OLTP load. As a result, the peak performance for all these tests was considered while ignoring the cycling issues. Detailed investigations are being carried out with the Quest support team to troubleshoot the cycling issues.


Topic See Page

Section A: Store solution component 73

Section B: Basic Backup solution component 79

Section C: Advanced Backup solution component 81

Section D: Basic Protect solution component 84

Section E: Advanced Protect solution component using EMC MirrorView and Oracle Data Guard

92

Section F: Advanced Protect solution component using EMC RecoverPoint

95

Section G: Test/Dev solution component using EMC 98




SnapView clone

Section H: Backup Server solution component 101

Section I: Migration solution component 103




Section A: Store solution component

Overview of the Store solution component

The Store solution component was designed as a set of performance measurements to determine the bounding point of the solution stack in terms of performance. A reasonable amount of fine tuning was performed in order to ensure that the performance measurements achieved were consistent with real-world, best-of-breed performance.

Test procedure The following procedure was used to validate the Store solution component:

Step Action

1 Close all the Benchmark Factory agents that are running.

2 Restart all the client machines.

3 Stop all the database instances.

4 Initiate the Benchmark Factory console and agents on the client machines.

5 Start the Benchmark Factory job.

6 Monitor the progress of the test.

7 Allow the test to finish.

8 Capture the results.




Test results

Summary The testing was conducted on a four-node RAC cluster. We scaled the RAC one

node at a time, from one node up to four nodes. The summary of the test results is shown below.

Test results for 1 RAC node – 3 FC shelves RAID 5/RAID 1

The table below shows the best result of the test runs for the 1 RAC node.

Users 3800

Transactions per second (TPS) 197.92

Response time 0.2 The chart below shows the relationship between users, transactions per second (TPS), and response time.

TPS / Response Time

0

50

100

150

200

250

2000 2500 3000 3500

User Count

TPS

0

2

4

6

8

10

Res

pons

e Ti

me

TPS

Response Time




Test results for 2 RAC nodes – 3 FC shelves RAID 5/RAID 1


Users 6400


Response time 0.537 The chart below shows the relationship between users, TPS, and response time.






Users 8400








Users 10000



TPS / Response Time

0

100

200

300

400

500

600

4000 5000 6000 7000 8000 9000 10000

User Count

TPS

0

100

200

300

400

500

600

Resp

onse

Tim

e

TPSResponse Time

Conclusion These results prove that the validated configuration produces the best performance

for this real-world configuration that the customer is likely to deploy.




Section B: Basic Backup solution component

Overview of the Basic Backup solution component

The Basic Backup solution component demonstrates that the validated configuration is compatible with Oracle Recovery Manager (RMAN) disk-to-disk backup. The backup tests were performance tests, where the performance of each node level was tested while creating an RMAN backup. The restore was a functionality test, but the amount of time required to perform the RMAN restore was also tuned and measured. The transactions that were restored and recovered were examined to ensure that there was no data loss.

Test configuration

The test configuration for the Basic Backup solution component was identical to the Store solution component.

Test procedure The following procedure was used to validate the Basic Backup solution component:

Step Action


2 Restart all the client machines and stop all the database instances.

3 Initiate the Benchmark Factory console and agents on the client machines.

4 Start the Benchmark Factory job.

5 Monitor the progress of the test.

6 Initiate RMAN backup at user load 3600 and monitor the progress.

7 Allow the test to finish.

8 Shut down the database and mount the database.

9 Perform the restore operation using RMAN and capture the observations.




Test results

Summary There was a moderate increase in response time and a moderate decrease in

transaction throughput when RMAN was initiated at user load 3500. Apart from that, TPS to user-load scaling was linear and the highest TPS of 444.44 was achieved at user load 8500.

Test results The RMAN backup started at user load 5000 and ended at user load 5200. The table

below shows the best result of the test runs. Users 5000 5200

Transactions per second (TPS) 58.32 273.35

Response time 79.699 0.063 The chart below shows the relationship between users, TPS, and response time.

Conclusion RMAN provides a reliable high-performance backup solution for the validated

configuration. However, the time required to restore the database is significant.




Section C: Advanced Backup solution component

Overview of Advanced Backup solution component

The Advanced Backup solution component demonstrates that the validated configuration is compatible with CLARiiON SnapView using Replication Manager. The backup tests were performance tests. The performance of each node level was tested while performing hot backup using SnapView snapshot with Replication Manager. The restore was a functionality test. The amount of time required to perform the SnapView restore was tuned and measured

Test Type Test Description

Performance 1 hot backup using SnapView snapshot with Replication Manager

Functional Restore and recover from a SnapView snapshot using Replication Manager

Test configuration

The test configuration for the Advanced Backup solution component was identical to the Store solution component.

Test procedure The following procedure was used to validate the Advanced Backup solution

component:

Step Action

1 Configure Replication Manager.

2 Register the production hosts, mount hosts, and storage in Replication Manager.

3 Create the application set in Replication Manager for the database to be replicated.

4 Create a job in the Replication Manager console to take the SnapView snapshot.


6 Close the Benchmark Factory console.

7 Restart the Benchmark Factory console and agents.

8 Stop and restart the database instances.

9 Start the Benchmark Factory test with the user load ranging from 4000 to 10000.

10 When the user load reaches iteration 5500, take a snapshot of the database by running the job in the Replication Manager console.




11 Monitor the performance impact on the production database.

12 When the Benchmark Factory test is complete, capture the results.

13 Shut down the database.

14 Stop and disable the ASM instances.

15 Dismount the data diskgroups.

16 Restore the database using Replication Manager.

17 Recover the database.

18 Capture the time taken to restore the database.

Test results

Performance test results

The snapshot job was initiated at user load 5500. For the optimum reported iteration, the TPS peaked at 9100 users. This was the highest user count that had a response time of less than 2 seconds. The table below shows the best result of the test runs.

Users 9100


Response time 0.129 The chart below shows the relationship between the number of users, TPS, and response time.




Functional test results

This test measured the time required to perform a full restore and recovery from a hot backup taken using a CLARiiON SnapView snapshot with Replication Manager. The backup created in the performance test above was used as the source for this restore.

Conclusion The CLARiiON SnapView feature works with Oracle RAC 11g/10g for our validated

configuration and can be performed successfully using Replication Manager. In most test runs, a very slight performance hit was observed during backup. However, this was temporary and the performance recovered to the expected levels within a short span of time. The restore of a SnapView hot backup is faster than an RMAN disk-to-disk restore.




Section D: Basic Protect solution component

Overview The Basic Protect solution component was designed to test the disaster recovery

functionality already built into the Oracle RAC of the validated configuration. The Basic Protect solution component assumes that there are no additional costs to the customer in terms of software or hardware. Oracle Data Guard was used for the Basic Protect solution.

Test configuration

The test configuration for the Basic Protect solution component was identical to the Store solution component.

Functional validation only

Only functional validation was done for the Basic Protect solution component. No tuning or performance measurements were carried out.

Test procedure The following procedure was used to validate the use of Data Guard with the

validated configuration:

Step Action

1 Configure Data Guard for Maximum Performance mode.

SQL> select PROTECTION_MODE, PROTECTION_LEVEL, DATABASE_ROLE from v$database;

PROTECTION_MODE PROTECTION_LEVEL DATABASE_ROLE

-------------------- -------------------- ----------------

MAXIMUM PERFORMANCE MAXIMUM PERFORMANCE PHYSICAL STANDBY

2 Enable “automatic archival” on the database.

SQL> ALTER DATABASE FORCE LOGGING;

3 Place the primary database in “Force Logging” mode.

4 Set the initialization parameters for the primary database. a) Create a parameter file from the spfile used by the primary database. b) Use the following commands to create the pfile “initmterac4.ora” for

the primary database: sqlplus / as sysdba;

Create pfile=’/u02/DataGuard/initmterac4.ora’ from spfile;

5 Modify the pfile “initmterac4.ora” for the Data Guard configuration as shown below:

## *.log_archive_dest_1='LOCATION=+SRCARCH/mterac4/'




**********************************************************************

ADD THE FOLLOWING FOR PRIMARY DATABASE DATAGUARD CONFIG

**********************************************************************

db_unique_name=mterac4

log_archive_config=’dg_config=(mterac4,mterac4_sb)’

log_archive_dest_1=’LOCATION=+SRCARCH/mterac4/

VALID_FOR=(ALL_LOGFILES,ALL_ROLES)

DB_UNIQUE_NAME=mterac4'

log_archive_dest_2=’service=mterac4_sb

VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)

DB_UNIQUE_NAME=mterac4_sb'

log_archive_dest_state_1=enable


fal_server=mterac4_sb

fal_client=mterac4

db_file_name_convert='+srcdata/mterac4/','+srcdata/mterac4/'

log_file_name_convert=’+srclog1/mterac4/’,’+srclog1/mterac4/’,’+srclog2/mterac4/’,’+srclog2/mterac4’

standby_file_management=auto

6 Modify the spfile parameters using the parameter file “initmterac4.ora”:

SQL>Create spfile=’+SRCDATA/mterac4/spfilemterac4.ora’ from pfile=/u02/DataGuard/initmterac4.ora’;

7 Copy a complete set of database datafiles to the destination CLARiiON LUNs using RMAN.

8 Create the standby control file for the standby database, then open the primary database to user access.

SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/u02/DataGuard/mterac4_sb.ctl';

SQL> ALTER DATABASE OPEN;

9 Create the parameter file for the standby database. The parameter values for the standby database are the same as the primary database.

10 Create and modify the “/u02/DataGuard/initmterac4_sb.ora” file as follows:




db_unique_name=mterac4_sb

log_archive_config=’dg_config=(mterac4,mterac4_sb)’

log_archive_dest_1=’LOCATION=+SRCARCH/mterac4/

VALID_FOR=(ALL_LOGFILES,ALL_ROLES)

DB_UNIQUE_NAME=mterac4_sb'

log_archive_dest_2=’service=mterac4

VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)

DB_UNIQUE_NAME=mterac4'



fal_server=mterac4

fal_client=mterac4_sb

db_file_name_convert='+srcdata/mterac4/','+srcdata/mterac4/'

log_file_name_convert=’+srclog1/mterac4/’,’ +srclog1/mterac4_sb/’,’+srclog2/mterac4/’,’+srclog2/mterac4_sb/’

standby_file_management=auto

*.control_files=’/u02/DataGuard/mterac4_sb.ctl’

11 Modify the “tnsnames.ora” file on all primary database nodes and standby database nodes to include all primary and standby net service names. This allows transparent failover.

12 Create a password file.

cd /u01/app/oracle/product/10.2.0/db_1/dbs

mv orapwmterac41 orapwmterac41_old




orapwd file=orapwmterac41 password=nasadmin







orapwd file=orapwmterac4_sb1 password=nasadmin




Important The password for the SYS user must be identical on every node (both primary and standby) for successful transmission of the redo logs.

12 Create the following directory in all standby nodes: /u01/app/oracle/product/10.2.0/db_1/admin/mterac4_sb

13 Create the following directories for each “/u01/app/oracle/product/10.2.0/db_1/admin/mterac4_sb” directory in all standby nodes: • adump • bdump • cdump • udump

14 Register the standby database and database instances with the Oracle Cluster Registry (OCR) using the server control utility.

srvctl add database -d mterac4_sb -o /u01/app/oracle/product/10.2.0/db_1

srvctl add instance -d mterac4_sb -i mterac4_sb1 -n mteoradb9




15 Update the value of ORACLE_SID on all standby nodes (mteoradb9, mteoradb10, mteoradb11 and mteoradb12). This setting is contained in the “.bash_profile.bash” file for all Oracle users.

16 Copy the modified pfile from the standby database (initmterac4_sb.ora) to the standby database nodes.

17 Copy the parameter file and the standby control file to the standby database nodes.

Note Because the ASM file system was used to configure the database, no utilities were supported by Oracle to copy the files between ASM disk groups. The control files were kept under the OCFS2 file system




(/u02/DataGuard/mterac4_sb.ctl). To use Oracle’s utility DBMS_FILE_TRANSFER to copy the contents to ASM disk groups, open the DR database in write mode. However, this defeats the purpose of the standby database. The standby database should not be opened in write mode at the standby site during normal operation. See the initialization parameter file in the previous procedure for the control_files parameter.

18 Perform the following steps to bring up the database:

SQL> startup nomount;

ORACLE instance started.

Total System Global Area 7516192768 bytes

Fixed Size 2082856 bytes

Variable Size 1241515992 bytes

Database Buffers 6257901568 bytes

Redo Buffers 14692352 bytes

SQL> alter database mount;

Database altered.

SQL> show parameter control_files;

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

control_files string /u02/DataGuard/mterac4_sb.ctl

19 Create the standby redo log files on the standby database. Important The size of the current standby redo log files must exactly match the size of the current primary database online redo log files.

20 Start the managed recovery and realtime apply on the standby database. The statement includes the DISCONNECT FROM SESSION option so that redo apply ran in a background session.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

21 Verify the status of the existing logs on the primary and standby database. Primary database SQL> select GROUP#, THREAD#, SEQUENCE#, ARCHIVED, STATUS from v$log;

GROUP# THREAD# SEQUENCE# ARC STATUS




--------- ---------- ---------- --- ---------------

1 1 85 YES INACTIVE

2 1 86 NO CURRENT

3 2 51 YES INACTIVE

4 2 52 NO CURRENT

5 3 51 YES INACTIVE

6 3 52 NO CURRENT

7 4 51 YES INACTIVE

8 4 52 NO CURRENT

8 rows selected.

Secondary database SQL> select GROUP#, THREAD#, SEQUENCE#, ARCHIVED, STATUS from v$log;

GROUP# THREAD# SEQUENCE# ARC STATUS

--------- ---------- ---------- --- ---------------

1 1 85 YES CLEARING

2 1 86 YES CLEARING_CURRENT

3 2 51 YES CLEARING


5 3 51 YES CLEARING


7 4 51 YES CLEARING


8 rows selected.

22 Force a log switch to archive the current online redo log file on the primary database.

SQL> alter system archive log current;

System altered.

23 Archive the new redo data on the standby database. To verify whether the standby database received and archived the redo data, on the standby database, query the V$ARCHIVED_LOG view.

SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME

2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

24 Apply the new archived redo log files. To verify whether the archived redo log files were applied, on the standby database, query the V$ARCHIVED_LOG view.

SQL> SELECT SEQUENCE#,APPLIED FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

SEQUENCE# APP




---------- ---

46 YES

46 YES

46 YES

47 YES

47 YES

47 YES

48 YES

48 YES

48 YES

49 YES

49 YES

SEQUENCE# APP

---------- ---

49 YES

50 YES

50 YES

50 YES

51 NO

51 NO

51 YES

79 YES

80 YES

81 YES

82 YES

SEQUENCE# APP

---------- ---

83 YES

84 YES

85 NO

25 rows selected.

Note The last redo log is not cleared. This is normal as another log is being applied by the time the last redo log is cleared.




Test results

Summary The Basic Protect solution component was used for both sites. No attempt was

made to emulate a WAN connection as this round of testing included only functionality tests. The production and disaster recovery (DR) sites are directly connected.

Test results No tuning or performance measurements were carried out for the Basic Protect

solution component.

Conclusion This solution component uses tools provided by the operating system and database

server software to provide disaster recovery. This is a very effective solution component that uses the database server’s CPU, memory, and I/O channels for all operations relating to the disaster recovery configuration.




Section E: Advanced Protect solution component using EMC MirrorView and Oracle Data Guard

Overview of Advanced Protect solution component

The purpose of this test series was to test the disaster protection functionality of the validated configuration in conjunction with EMC MirrorView. The Advanced Protect solution component uses MirrorView to enhance the disaster recovery functionality of the blended FCP/NFS solution. The use of Oracle Data Guard to create a disaster recovery configuration is an established best practice. MirrorView/Asynchronous (MirrorView/A) over iSCSI is commonly used as a way of seeding the database for the Data Guard configuration. After the production database was copied to the target location, redo log shipping was then established using Data Guard. No attempt was made to emulate a WAN connection in this round of testing. The production site and the disaster recovery (DR) site were directly connected over a LAN. The advantages of MirrorView are: • The data can be replicated over a long distance. Local access is not required. • No downtime on the source database is required.

Test configuration

The test configuration for the Advanced Backup solution component was identical to the Store solution component.

Test procedure The following procedure was used to validate the Advanced Protect solution

component:

Step Action






6 When the user load reaches iteration 5000, run a script to mirror the LUNs.

7 Wait for the Benchmark Factory test to complete and for the mirrors to synchronize and to become consistent.

8 Fracture the mirrors.

9 Copy the standby control file and the parameter file that is created from




production to the target host.

10 Update the parameters at the target host.

11 Start the database in mount phase on the target host.

12 Do the remaining tasks to enable Data Guard to carry out redo log shipping.

13 Do the switchover and switchback.

Test results

Summary Testing showed that a Data Guard configuration could be successfully initialized (or

“seeded”) using MirrorView. However, performance was limited by Data Guard to only 8000 users and 415 TPS. Data Guard is a viable option for remote replication but clearly creates a performance penalty.

Test results TPS peaked at 8000 users. This was the highest user count with a response time of

less than 2 seconds. The table below shows the best result of the test runs.

Users 8000


Response time 0.191 The chart below shows the relationship between the number of users, TPS, and response time.




Conclusion This solution component enables the creation of a writeable copy of the production

database on the disaster recovery target, which allows this database to be used for operations such as backup, test/dev, and data warehouse staging. The solution component uses additional software components at the storage layer to enable disaster recovery, which frees up the database server’s CPU, memory, and I/O channels from the effects of these operations.




Section F: Advanced Protect solution component using EMC RecoverPoint

Overview of Advanced Protect solution component

The purpose of this solution component was to replicate a small RAC database from the production site and to bring up the replicated database at the target site as a single-instance database on a VMware host.


Only functional validation was done for the Advanced Protect solution component using EMC RecoverPoint with CLARiiON (CX) splitters. No tuning or performance measurements were carried out. EMC plans to carry out performance testing during the next test cycle.

Test configuration

The test configuration as shown in the image below was used to validate the Advanced Protect solution component.




Test procedure The following procedure was used to validate the functionality of RecoverPoint with

the Advanced Protect solution component:

Step Action

1 Create a small database using the listed LUNs at the production site.

2 Create consistency groups comprising of DATA and REDO LOG LUNs.

3 Establish the replication pairs for these LUNs.

4 Enable the consistency groups.

5 Verify that the replication starts successfully.

6 While the replication is in progress, create a table named “foo” at the source site and insert a row.

7 At the target site, access the latest system-defined bookmark image and discover the replicated LUNs on the VMware ESX server.

8 At the target site, map the LUNs discovered by the ESX server onto the VMware host using Raw Device Mapping (RDM). Note Detailed instructions for mapping the LUNs from an ESX server to a VMware host using RDM is available in Chapter 8: Virtualization > VMware ESX server > LUN discovery.

9 Once the LUNs are mapped to the VMware host, discover the corresponding PowerPath devices and ASM disks.

10 Mount the ASM disk groups and modify the pfile parameters so that the replicated database can be brought up as a single-instance database.

11 Bring up the single-instance database at the target site and verify that the entries inserted in table “foo” at the source site have been successfully replicated to the target site.

LUN designation

A single VMware host was used to bring up a single-instance database at the target site. The source LUNs were designated as follows: Name Number Type RAID Size Data 320 FC R5 2GB Log1 321 FC R5 1GB Arch 330 FC R5 2GB Log2 331 FC R5 1GB Flash 340 FC R5 2GB The target LUNs were designated as follows: Name Number Type RAID Size




Data replica 320 FC R10 2GB Log1 replica 321 FC R10 1GB Arch replica 330 FC R10 2GB Log2 replica 331 FC R10 1GB Flash replica 340 FC R10 2GB

Test results

Summary RecoverPoint remote replication was successfully tested for Oracle database

environments. The functionality required by the solution were fully validated.

Test results No tuning or performance measurements were carried out for the Advanced Protect

solution component. This will be done in future testing.

Conclusion EMC RecoverPoint with CX splitters can be used to successfully replicate a

database in the context of the validated configuration described in the reference architecture for this solution.




Section G: Test/Dev solution component using EMC SnapView clone

Overview of Test/Dev solution component

The Test/Dev solution component provides a rapid, high-performance method for copying a running Oracle 11g/10g database. The copy can then be used for testing and development purposes.

SnapView clones

EMC SnapView clone can be used to clone a running Oracle RAC 11g/10g production database for rapidly creating testing and development copies of this database. A number of best practices should be followed.

Archived log and flashback recovery area LUNs While cloning the LUNs, it is better to avoid cloning the archived log and flashback recovery area LUNs. These LUNs are usually large and are typically stored on LCFC or SATA II disks. The combination of these settings means that cloning these LUNs will take much longer than cloning the datafile LUNs. If required, you can configure a separate set of LUNs for the archived logs and flashback recovery area for the testing or development database. Since the test/dev database can be easily refreshed, you may choose to simply skip backing up these databases. In this case, a flashback recovery area is not required. In order for the test/dev database to perform a successful recovery, the archived logs from the production database should be accessible to the test/dev database. You must always move the archived log dump destination of the test/dev database to a new set of LUNs before opening this database for transactional use. Significant transactional I/O on the test/dev database could create an archived log. If you do not change the archive dump destination of the test/dev database, and it uses the same LUNs for storing the archived logs as the production database, the test/dev database could overwrite the archived logs of the production database. This could destroy the recoverability of the production database. This setting is contained in the pfile or spfile of the test/dev database in the parameters LOG_ARCHIVE_DUMP_DEST1 to LOG_ARCHIVE_DUMP_DEST10.

SyncRate While initializing the clone, the most important setting is the SyncRate. If you need the test/dev database to be created rapidly, specify the SyncRate option as “high”. This speeds up the synchronization process, but at the cost of a greater performance impact on the production database. If performance on the production database is your primary concern, specify the SyncRate option as “low”. Here is an example of the naviseccli command using the SyncRate option:

naviseccli –address 10.6.24.205 snapview –addclone –name lun0CloneGrp –luns 50

-SyncRate <high|medium|low|value> If you do not specify this option, then the default is set as “medium”.




Test configuration

The test configuration for the Test/Dev solution component was identical to the Store solution component. A storage group named “EMC Replication Storage” was created and populated with target LUNs for storing replicas. Replication Manager takes care of copying data from the source LUNs to the target LUNs. The time required to create the clone depends on the size of the source LUN. For this test, only the data and redo log LUNs were cloned. Because of the amount of time required, archived logs and flashback recovery area files were not cloned. Only the source archive and the flash LUNs were used to bring up the cloned database.

Testing procedure

The following procedure was used to validate the Test/Dev solution component:

Step Action

1 Configure Replication Manager.

2 Register the production hosts, mount hosts, and storage in Replication Manager.

3 Create the application set in Replication Manager for the database to be replicated.

4 Create a job in the Replication Manager console to create a SnapView clone.






10 When the user load reaches iteration 6000, take a snapshot of the database by running the job in the Replication Manager console. Monitor the performance impact on the production database.

11 Capture the results when the Benchmark Factory test completed.

12 Discover the PowerPath devices and ASM disks for the mount host (target database server).

13 Perform the mount and recovery of the replica (SnapView clone) on the mount host using Replication Manager.

14 Capture the time taken to recover the database.




Test results

Summary The SnapView clone was successfully validated. The performance impact of the

operation was minor. Performance was very similar to the baseline.

Test results The cloning job was initiated at user load 6000 and ended at user load 8000.

TPS peaked at 9100 users. This was the highest user count that had a response time of less than 2 seconds. The table below shows the best result of the test runs.

Users 8000 9100

Transactions per second (TPS) 309.32 476.17

Response time 16.478 0.121 The chart below shows the relationship between the number of users, TPS, and response time.

Conclusion The CLARiiON SnapView clone feature works with the validated configuration and

can be performed successfully through Replication Manager. In most test runs, a modest performance hit was observed during hot backup, however, this was temporary. The performance recovered to the expected levels after 10 to 15 minutes. This again depends on the size of the database.




Section H: Backup Server solution component

Overview The purpose of the Backup Server solution component is to offload the burden of

backup from the production database servers to a utility database server running within a VM. The Backup Server solution component is becoming more popular because of higher performance disk-to-disk snapshot technology (using Replication Manager). Oracle Database 11g RMAN has a CATALOG BACKUPPIECE command. This command adds backup pieces of information of the target database on disk to the production database RMAN repository. The backup pieces should be on the shared location. As long as backup pieces are accessible to both production and target databases, RMAN commands such as RESTORE and RECOVER behave transparently across different databases.

Test configuration

The test configuration for the Backup Server solution component was identical to the Store solution component.

Test procedure The following procedure was used to validate the Backup Server solution

component:

Step Action





5 Start the Benchmark Factory test and start ranging the user load from 4000 to 10000.

6 When the user load is at iteration 5600, initiate cloning of the production database with Replication Manager.

7 Start the ASM instance in the target server, mount the database, and run the RMAN backup.

8 Place the RMAN backup pieces in a shared flashback recovery area. The backup pieces should be accessible to both the target and production servers.

9 Catalog the backup pieces using the CATALOG BACKUPPIECE command within RMAN in the production server.

10 Shut down the production server.




11 Perform restore and recovery on the production server of the backup taken on the target server.

Test results

Summary Minimal performance impact was observed on the production server when using the

backup server approach to backup. Basically, the load relating to backup was offloaded from the production server. The only performance impact occurred when the snapshot was taken.

Test results for 3 FC shelves/ASM with user-defined pools

The table below shows the best result of the test runs.

Users 5600



Conclusion RMAN provides a reliable high-performance backup server solution for the Oracle

RAC 11g/10g configuration. However, the time required to restore the database is significant. The restore operation for a 2000-warehouse database typically takes around one and half hours.




Section I: Migration solution component

Overview The Migration solution component demonstrates that EMC Replication Manager can

be used to migrate an Oracle 11g/10g database mounted on FCP/ASM to a target database mounted on NFS with minimum performance impact and no downtime of the production database.

Test objectives This test was just a functionality validation of the migration of Oracle 11g/10g

database from a SAN to NAS configuration. The performance impact on the production database during online migration was not validated.

Test configuration

The test configuration for the Migration solution component was identical to the Store solution component.

Test procedure The following procedure was used to validate the Migration solution component:

Step Action

1 Using EMC Replication Manager, perform a consistent backup of the running production database on the CLARiiON using a CLARiiON SnapView snapshot.

2 Mount (but do not open) this backup on the migration server, in this case a VMware virtual machine (VM) (a physically booted server would also work). The NFS target array is also mounted on the migration server.

3 Using Oracle Recovery Manager (RMAN), make a backup of this database onto the target site. This backup is performed as a database image, so that the datafiles are written directly to the target NFS mount.

4 Switch the migration server to the new database that has been copied by RMAN to the NFS mount.

5 Set the target database in Data Guard continuous recovery mode, and use Data Guard log ship/log apply to catch the target database up to the production version.

6 Once the target database is caught up to production, use Data Guard failover to retarget to the target database. If appropriate networking configuration is performed, clients will see no downtime when this operation occurs.




Test results

Summary We were able to validate the functionality of migrating an online production Oracle

database from FCP to NFS. This was accomplished with minimal performance impact and no downtime on the production server.

Test results The test showed that the impact on the production database was identical to the

backup server, which is contained in Chapter 7: Testing and Validation > Section H: Backup Server solution component.

Conclusion The ability to migrate a SAN to NAS database is a frequent request from the

customer. Customers are forced to switch from SAN to NAS or NAS to SAN based on ever-changing requirements. This solution proves that EMC Replication Manager can be a very effective application to achieve this goal with ease and no impact or downtime to the production database.

Chapter 8: Virtualization




Overview

Introduction to virtualization

Virtualization lets a customer run multiple virtual machines on a single physical machine, sharing the resources of that single computer across multiple environments. Different virtual machines can run different operating systems and multiple applications on the same physical computer. The VMware virtualization platform is built on a business-ready architecture. Customers can use software such as VMware vSphere and VMware ESX to transform or “virtualize” the hardware resources of an x86-based computer - including the CPU, RAM, hard disk, and network controller - to create a fully functional virtual machine that can run its own operating system and applications just like a “real” (or physical) computer. Server virtualization offers energy efficiencies, cost savings, and better management of service level agreements. VMware ESX abstracts server processor, memory, storage, and networking resources into multiple virtual machines. By doing so, it can dramatically improve the utilization of these resources.

Using virtualization

Virtualized Oracle database servers were used as targets for test/dev, backup, and disaster recovery for this solution. These servers are more conveniently managed as virtual machines than as physically booted Oracle database servers. The advantages of consolidation, flexible migration and so forth, which are the mainstays of virtualization, apply to these servers very well. A single VMware Linux host was used as the target for test/dev, backup, and disaster recovery. For test/dev, the target database was brought up as a single-instance database on the VMware host. Similarly, the standby database for disaster recovery was a single-instance database running on a VMware host. This chapter provides procedures and guidelines for installing and configuring the virtualization components that make up the validated solution scenario.


Topic See Page

Advantages of virtualization 106

Considerations 106

VMware infrastructure 106

Virtualization best practices 109

VMware ESX server 110

VMware and NFS 111




Advantages of virtualization

Advantages Some advantages of including virtualized test/dev and disaster recovery (DR) target

servers in the solution are: • Consolidation • Flexible migration • Cloning • Reduced costs

Considerations

Virtualized single-instance Oracle only

Due to the requirement for RAC qualification, presently there is no support for Oracle 11g and 10g RAC servers on virtualized devices. For this reason, EMC does not publish such a configuration as a supported and validated solution. However, the use of Oracle Database 11g and 10g (in single-instance mode) presents far fewer support issues.

VMware infrastructure

Setting up the virtualized utility servers

For details on setting up the virtualized utility servers, see Chapter 6: Installation and Configuration > Task 13: Set up the virtualized utility servers > Setting up the virtualized utility servers.

Virtualization best practices

VMotion storage requirements

You must have a common storage network configured on both source and target ESX servers to perform VMotion. The network configuration including the vSwitch names should be exactly the same. The connectivity to the LUNs on the back-end storage from the ESX servers should also be established in the same way.

ESX servers must have identical configuration

All ESX servers must have an identical configuration, other than the IP address for the VMkernel port.




Dedicated private connection

When NFS connectivity is used, it is a best practice to have a dedicated private connection to the back-end storage from each of the VMs. We did the following: • Assigned four NICs (one NIC for each VM) on the ESX server • Assigned private IPs to the NICs • Set up the connectivity from these four NICs to the Data Movers of the back-end

storage using a Dell PowerConnect switch

NFS mount points

If the Oracle database files sit on NFS storage, the NFS share should be mounted as a file system within the Linux guest VM using /etc/fstab. This can deliver vastly superior performance when compared to storing Oracle database files on virtual disks that reside on an NFS share and are mounted as NFS datastores on an ESX server.

VMware ESX server

Typical storage network configuration

The image below contains some detail from the Configuration -> Network tap on the VMware vSphere vCenter server. The storage network configuration is shown. We used two physical NICs to support the storage network. This provides physical port redundancy, as well as link aggregation and load balancing across the NICs. Ports are provided for both the NFS mounts on the VMs, as well as the NFS mounts on the ESX servers.




Storage configuration

In the image below, the VMkernel Storage Network is being used to store the files for the VMs (through NFS). The storage pane shows that the NFS-mounted volume “vm” is where these files are stored.

LUN discovery LUNs can be discovered on an ESX server in two ways:

• The first method uses Virtual Machine File System (VMFS). • The second method uses Raw Device Mapping (RDM). EMC recommends using RDM for discovering the LUNs on the ESX server because RDM provides better disk I/O performance and also supports VMotion.

VMware and NFS

NFS mounts in ESX

NFS is a viable storage option for VMware ESX. It provides a simple and manageable storage networking alternative to FCP or iSCSI. NFS does not require the ESX servers to run a clustered file system (in this case VMFS). For the utility servers used in this solution, NFS was used to store the OS images for the VMs.

Chapter 9: Backup and Restore




Overview

Introduction to backup and restore

A thoughtful and complete backup strategy is an essential part of database maintenance in a production environment. Data backups are an essential part of any production environment. Regardless of the RAID protection level, hardware redundancy, and other high-availability features present in EMC Celerra storage arrays, conditions exist where you may need to be able to recover a database to a previous point in time. This solution used EMC SnapView to free up the database server’s CPU, memory, and I/O channels from the effects of operations relating to backup, restore, and recovery.

Scope This section covers the use of SnapView snapshots to perform backup and restore

operations on Oracle RAC database servers. Important note on scripts The scripts provided assume that the passwordless authentication is set up using ssh between the oracle user account and the Celerra Control Station. Passwordless authentication allows the oracle user account to issue commands to the Control Station within a script. Instructions on how to accomplish this can be found in Chapter 6: Installation and Configuration > Task 10: Enable passwordless authentication using SSH.


Topic See Page

Section A: Backup and restore concepts 110

Section B: Backup and recovery strategy 112

Section C: Physical backup and restore 116

Section D: Replication Manager in Test/Dev and Advanced Backup solution components

119




Section A: Backup and restore concepts

Physical storage backup

A full and complete copy of the database to a different physical media.

Logical backup A backup that is performed using the Oracle import/export utilities.

The term “logical backup” is generally used within the Oracle community.

Logical storage backup

Creating a backup using a logical image is referred to as a logical storage backup. A logical storage backup is a backup that does not physically exist. Rather, it consists of the blocks in the active file system, combined with blocks in a SavVol, an area where the original versions of the updated blocks are retained. The effect of a logical storage backup is that a view of the file system as of a certain point in time can be assembled. Unlike a physical storage backup, a logical storage backup can be taken very rapidly, and requires very little space to store (typically a small fraction of the size of a physical storage backup). Important Taking logical storage backups is not enough to protect the database from all risks. Physical storage backups are also required to protect the database against double disk failures and other hardware failures at the storage layer.

Flashback Database

The Oracle Flashback Database command enables you to restore an Oracle database to a recent point in time, without first needing to restore a backup of the database.

EMC SnapView snapshot

The EMC SnapView snapshot allows a database administrator to create a point-in-time copy of the database that can be made accessible to another host or simply held as a point-in-time copy for possible restoration.




Advantages of logical storage

Recovery from human errors

Logical backup protects against logical corruption of the database, as well as accidental file deletion, and other similar human errors.

Frequency without performance impact

The logical storage operation is very lightweight and, as a result, a logical storage backup can be taken very frequently. Most customers report that they cannot perceive the performance impact of this operation because it is so slight.

Reduced MTTR Depending on the amount of data changes, restoring from a logical storage backup

can occur very quickly. This dramatically reduces mean time to recovery (MTTR) compared to what can be achieved restoring from a physical backup.

Less archived redo logfiles

Due to the high frequency of backups, a small number of archived redo log files need to be applied if a recovery is needed. This further reduces mean time to recovery.




Section B: Backup and recovery strategy

Use both logical and physical backup

Logical backup In the context of this document, the term “logical storage backup” is distinguished from the term “logical backup,” which is generally used within the Oracle community and which is a backup performed using the import/export utilities. A logical storage backup is a backup that does not physically exist. Rather, it consists of the blocks in the active LUN, combined with blocks in a SavVol, an area where the original versions of updated blocks are retained. The effect of a logical storage backup is that you can assemble a view of the LUN as of a certain point in time. A logical storage backup is distinguished also from a physical storage backup, which is a full and complete copy of the database objects made to different physical media. Unlike a physical storage backup, a logical storage backup can be taken very rapidly, and requires very little space to store (typically a small fraction of the size of a physical storage backup). At the center of the logical backup approach is the EMC CLARiiON SnapView feature, which uses the Navisphere SnapView command. This command allows a database administrator to capture a logical storage backup of an entire LUN, or with consistency technology, a group of LUNs. The best practice for the backup of Oracle Database 11g/10g is to perform approximately six logical storage backups per day, at four-hour intervals, using CLARiiON SnapView. To facilitate the ability to recover smaller granularities than the datafile (a single block for example), you should catalog all the SnapView snapshots backups within the RMAN catalog.

Physical backup As logical backups do not protect you from hardware failures (such as double-disk failures), you should also perform one physical backup per day, typically during a period of low user activity. For this purpose, EMC recommends RMAN using an incremental strategy, if the database is larger than 500 GB, and using a full strategy otherwise. Further, EMC recommends that the RMAN backup be to a SATA II disk configuration rather than to tape.

Reduced mean time to recovery

Using a strategy that combines physical and logical backups optimizes the mean time to recovery. In the event of a fault that is not related to the hardware, you can restore instantly from a SnapView snapshot. According to Oracle, approximately 90 percent of all restore/recovery events are not related to hardware failures, but rather to user errors, such as deleting a datafile or truncating a table. Further, the improved frequency of backups over what can be achieved with a blended physical backup strategy means that you have fewer logs to apply, thereby improving mean time to recovery (MTTR). Even in the case where you need to restore from physical backup, the use of SATA II disk will improve restore time.




Logical storage backup using EMC SnapView and EMC Replication Manager

Overview of SnapView

A logical storage backup consists of virtual copies of all LUNs being used to store datafiles. This is enabled by EMC CLARiiON SnapView. Using EMC consistency technology, multiple LUNs can be used to store an automated storage management (ASM) disk group and snapshots of all of those LUNs can be created in a consistent manner.

SnapView snapshot

A SnapView snapshot is a virtual point-in-time copy of a LUN. This virtual copy is assembled by using a combination of data in the source LUN, and the “before” images of updated blocks that are stored on the CLARiiON target array in the Reserved LUN Pool (RLP). Note In Replication Manager, the RLP is referred to as the “snap cache”. We will adhere to Replication Manager terminology and use the term “snap cache”.

Multiple restore points using EMC SnapView

The following image compares backup using SnapView to conventional backup over a typical 24-hour period.

StorageDatabase

Tape

Celerra SnapSureCLARiiON SnapView

Midnight 8:00 a.m. Noon 4:00 p.m. 8:00 p.m. Midnight4:00 a.m.

AFTER

Conventionalbackup

BEFO

RE

Midnight Midnight

Rapid restore and recovery using EMC SnapView

The following image compares restore and recovery using SnapView to conventional backup over a typical 24-hour period.

Restore (move data files)Backup

Recovery (apply logs)

Tapebackup Multiple hours

Data loss

BEFO

RE

Backup time Recovery time

Minutes

Backup time Recovery time

AFTER

Celerra SnapSureCLARiiON SnapView




Replication Manager

EMC Replication Manager automates the creation and management of EMC disk-based point-in-time replicas. Replication Manager integrates with the Oracle database server and provides an easy interface to create and manage Oracle replicas.

Logical storage process

A typical backup scheme would use six logical storage backups per day, at four-hour intervals, combined with one physical storage backup per day. The procedure described next can be integrated with the Oracle Enterprise Manager job scheduling process or cron. This can be used to execute a logical storage backup once every four hours.

Logical storage backup using Replication Manager

The following table outlines the steps used to perform a backup using SnapView snapshots:

Step Action

1 Create a Replication Manager job for SnapView snapshot: a. Select Jobs, right-click and select New Job. b. To go to the Job Name and Settings screen, click Next. c. Enter: job name, replication source, replication technology, and number of replicas to be created, then click Next. d. Select the mount options, then click Next.

Note: If necessary, you can select the mounting options at a later point in time.

e. At the Completing the Job wizard screen, click Finish.

2 Execute a Replication Manager job: a. In Replication Manager, select Jobs from the navigation panel. b. Select a job, right-click and select Run to execute the job. c. To confirm that you want to run the job, click Yes. d. When the Running Job progress bar reaches 100%, click Close.

The status of the job displays as “Successful”.

3 To verify the Snapshot Session information, select the appropriate item under Storage Services. Note: You can also use Navisphere to view the snapshot sessions that were created using Replication Manager.




Best practice During backup, EMC recommends that you:

• Switch the log files • Archive all the log files • Back up the control file The instructions on how to carry out these operations are beyond the scope of this document.




Section C: Physical backup and restore

Physical backup using Oracle RMAN

RMAN and Celerra

Physical backup of the production array can be accomplished using Oracle RMAN. The target was an NFS mount on the Celerra. The backup target is typically SATA or LCFC disks on the target array. If tape is used with a product that includes a media management layer, such as EMC NetWorker®, Oracle Secure Backup must be used. Normal RMAN semantics apply to this backup method. This is thoroughly covered on the Oracle Technology Network website and will not be included in this document.

RMAN backup script: rmanbkp.bash

Run the following script from the database server to carry out physical backup of a Celerra array using Oracle RMAN:

[oracle@mteoradb55 ~]$ cat rmanbkp.bash

#!/bin/bash

. ~/.bash_profile

. /cygdrive/c/common/initialize.bash

echo "This is oracle_cold_backup_database.bash"

echo "rman"

echo "connect target /"

echo "backup database plus archivelog;"

echo "exit"

rman <<EOF2

connect target /

backup database plus archivelog;

exit

EOF2

echo "Now exiting oracle_cold_backup_database.bash"

exit

[oracle@mteoradb55 ~]$ .




System output of rmanbkp.bash

The output our system produces after running rmanbkp.bash is shown below:

[oracle@mteoradb55 ~]$ . rmanbkp.bash

This is rmanbkp.bash

Starting RMAN Backup

Recovery Manager: Release 10.2.0.3.0 - Production on Fri Mar 7 00:49:52 2008

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to target database: MTERAC16 (DBID=2284367788)

RMAN> backup database;

Starting backup at 07-MAR-08

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: sid=4512 instance=mterac161 devtype=DISK

channel ORA_DISK_1: starting full datafile backupset

channel ORA_DISK_1: specifying datafile(s) in backupset

input datafile fno=00008 name=+DATA/mterac16/datafile/test.265.645938379









input datafile fno=00017




name=+DATA/mterac16/datafile/test.274.645939115


input datafile fno=00002 name=+DATA/mterac16/datafile/undotbs1.258.645935993


input datafile fno=00003 name=+DATA/mterac16/datafile/sysaux.257.645935993

input datafile fno=00001 name=+DATA/mterac16/datafile/system.256.645935993



input datafile fno=00004 name=+DATA/mterac16/datafile/users.259.645935993

channel ORA_DISK_1: starting piece 1 at 07-MAR-08

channel ORA_DISK_1: finished piece 1 at 07-MAR-08

piece handle=/u06/MTERAC16/backupset/2008_03_07/o1_mf_nnndf_TAG20080307T005008_3x1oz3t8_.bkp tag=TAG20080307T005008 comment=NONE

channel ORA_DISK_1: backup set complete, elapsed time: 01:21:26

Finished backup at 07-MAR-08

Starting Control File and SPFILE Autobackup at 07-MAR-08

piece handle=/u06/MTERAC16/autobackup/2008_03_07/o1_mf_s_648699094_3x1to7tb_.bkp comment=NONE

Finished Control File and SPFILE Autobackup at 07-MAR-08

RMAN>

End of RMAN backup rmanbkp.bash




Section D: Replication Manager in Test/Dev and Advanced Backup solution components

Overview The Test/Dev and Advanced Backup solution components are integrated with EMC

Replication Manager. This has significant advantages in that Replication Manager provides a layered GUI application to manage these processes. This includes a scheduler so that the jobs can be run on a regular basis. Replication Manager, however, introduces a few issues that are covered in this section.

Oracle home location

Currently, Replication Manager does not support ASM and Oracle having separate Oracle homes. This may be confusing, because the Oracle installation guide presents an installation in which ASM is located in its own home directory. Important If you choose to use Replication Manager for storage replication management, install Oracle and ASM in the same home directory.

Dedicated server process

Replication Manager cannot create an application set when connected to the target database using SHARED SERVER. Replication Manager requires a dedicated server process. In the TNSNAMES.ORA file, you must modify the value of SERVER as shown below to connect to the target database. This is only needed for the service that is used for the Replication Manager connection.

# tnsnames.ora Network Configuration File: /u01/app/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora

# Generated by Oracle configuration tools.

MTERAC211 =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = mteoradb67-vip)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = mterac21)

(INSTANCE_NAME = mterac211)

)

)

Chapter 10: Data Protection and Replication




Overview

Introduction This solution provides options to create local and remote replicas of application data

that are suitable for testing, development, reporting, and disaster recovery and many other operations that can be important in your environment.


Topic See Page

Section A: Basic Protect using Oracle Data Guard 121

Section B: Advanced Protect using EMC MirrorView and Oracle Data Guard

122

Section C: Advanced Protect using EMC RecoverPoint 127




Section A: Basic Protect using Oracle Data Guard

Overview Oracle Data Guard is used with the Basic Protect solution component.

For best practices on Oracle Data Guard configuration, refer to the Oracle documentation on this subject.

EMS CLARiiON SAN Copy and Oracle Data Guard

The best practice for disaster recovery of an Oracle Database 11g/10g over NFS is to use the CLARiiON SAN Copy for seeding the disaster recovery copy of the production database, and then to use the Oracle Data Guard log transport and log apply services. The source of the database used for seeding the disaster recovery site can be a hot backup of the production database within a Celerra SnapSure checkpoint. This avoids any downtime on the production server relative to seeding the disaster recovery database. The configuration steps for shipping the redo logs and bringing up the standby database are accomplished using Oracle Data Guard. The Data Guard Failover operation was performed in MAXIMUM AVAILABILITY mode. For best practices on Oracle Data Guard configuration, refer to the Oracle documentation on this subject. The following image illustrates the setup for disaster recover using CLARiiON SAN Copy and Oracle Data Guard.




Section B: Advanced Protect using EMC MirrorView and Oracle Data Guard

Overview The use of Oracle Data Guard to create a disaster recovery configuration is an

established best practice. EMC MirrorView Asynchronous (MirrorView/A) over iSCSI is commonly used as a way of seeding the database for the Data Guard configuration. Once the production database has been copied to the target location, then redo log shipping can be established using Data Guard. The use of MirrorView over iSCSI requires specific network configuration. Various means can be used to bridge an iSCSI network over a WAN connection so that the data on an iSCSI network can be transmitted over long distances. The advantages of MirrorView are: • The data can be replicated over a long distance. Local access is not required. • No downtime on the source database is required. Note EMC assumes that you have established a mechanism to transmit the data from the source to the target array over an IP network using the iSCSI protocol.

MirrorView mirroring prerequisites

Before starting the mirroring procedure: • The source and target arrays must be in the same CLARiiON domain. − You must designate the source CLARiiON as the master and the target

CLARiiON as a domain member. Once this is done, both the source and target arrays appear in the same Navisphere GUI and can be managed together.

− This is mandatory for MirrorView mirroring to be established. • Reserved LUN pool LUNs are required for MirrorView mirroring. − You should configure a large number of small LUNs for best results. We

configured 20 LUNs of 10 GB in size.

Seeding the Oracle database

Use MirrorView to seed the Oracle database for the Data Guard configuration as described in the following table:

Step Action

1 Create a consistency group on the source array with the following command:

[root@mteoradb1 db_root]# naviseccli -h 10.6.24.205 mirror –async \

-creategroup –name ConsistentDBGroup –o

2 Create a mirror group for the source LUNs, for example:

[root@mteoradb59 ~]# naviseccli -h 10.6.27.65 mirror -async -create -name LUN1_LOG1_mirror -lun 1 -




requiredimages 1

Warning Make sure you are finished enabling paths among all arrays. If not, exit and do so.

3 Verify that all the mirror groups were created successfully using the following command:

[root@mteoradb59 ~]# naviseccli -h 10.6.27.65 mirror -async -list

4 Use the following script to add the target LUNs to the mirror groups and also to the consistency group: Code Listing 22: MirrorView/A Data Guard seeding script [root@mteoradb59 db_root]# cat mirror_new.bash

echo "This is mirror.bash"

DATA_LUNS="3 4 5 6 7 8"

LOG_LUNS="1 2"

SPA=10.6.27.65

echo "Add the target LOG LUNs for mirroring"

for i in ${LOG_LUNS}

do

echo "Now adding lun LUN${i} of target Clarion to mirror"

naviseccli -address ${SPA} mirror -async -addimage -name LUN${i}_LOG${i}_mirror -arrayhost 10.6.27.138 -lun 5${i} -recoverypolicy auto -syncrate high

done

echo "Add the target DATA LUNs for mirroring"

for i in ${DATA_LUNS}

do

echo "Now adding lun LUN${i} of target Clarion to mirror"

naviseccli -address ${SPA} mirror -async -addimage -name LUN${i}_DATA_mirror -arrayhost 10.6.27.138 -lun 5${i} -recoverypolicy auto -syncrate high

done

echo "Now adding mirror for LOG LUNS to consistent group."

for i in ${LOG_LUNS}

do

naviseccli -address ${SPA} mirror -async -addtogroup -name ConsistentDBGroup -mirrorname




LUN${i}_LOG${i}_mirror

done

echo "Now adding mirror for DATA LUNS to consistent group."

for i in ${DATA_LUNS}

do

naviseccli -address ${SPA} mirror -async -addtogroup -name ConsistentDBGroup -mirrorname LUN${i}_DATA_mirror

done 26644394 echo "Now exiting mirror.bash" Code Listing 23: The output from the MirrorView/A script [root@mteoradb59 db_root]# ./mirror_new.bash

This is mirror.bash

Add the target LOG LUNs for mirroring

Now adding lun LUN51 of target Clarion to mirror


Add the target DATA LUNs for mirroring







Now adding mirror for LOG LUNS to consistent group.

Now adding mirror for DATA LUNS to consistent group.

Now exiting mirror.bash

5 Verify the mirroring status by executing the following command:

[root@mteoradb59 ~]# naviseccli -h 10.6.27.65 mirror -async \

-list -images | grep Progress

Synchronizing Progress(%): 100











6 Once the mirrors are synchronized, fracture them using the following command:

[root@mteoradb59 ]# naviseccli -h 10.6.27.65 mirror -async \

-fracturegroup -name ConsistentDBGroup –o Note The mirror LUNs will become available for I/O only after they are fractured.

Once the seeding of the database is complete, the remaining tasks for shipping the

redo logs can be performed.

Shipping the redo logs using Data Guard

The configuration steps for shipping the redo logs and bringing up the standby database are accomplished using Oracle Data Guard. The semantics are covered thoroughly on the Oracle Technology Network website and will not be included here.

Data Guard failover operation

The Data Guard failover operation is performed in MAXIMUM AVAILABILITY mode using the following steps:

Step Action

1 Shut down all the database instances at the production/primary site.

MTERAC71> shutdown abort

ORACLE instance shut down.







2 Issue the following commands to change the standby database to primary:

[MTERAC7 – Production Database & MTERAC7-SB – Standby Database]

MTERAC7_SB>SELECT THREAD#, LOW_SEQUENCE#, HIGH_SEQUENCE# FROM $ARCHIVE_GAP;

no rows selected




MTERAC7_SB>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH;

Database altered.

MTERAC7_SB>ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;

Database altered.

MTERAC7_SB>shutdown immediate;

ORA-01109: database not open

Database dismounted.


mteora7_sb>set sqlprompt ‘NEW_PRIMARY> ‘

NEW_PRIMARY>startup

ORACLE instance started.

Total System Global Area 171966464 bytes

Fixed Size 2069712 bytes

Variable Size 113249072 bytes

Database Buffers 50331648 bytes

Redo Buffers 6316032 bytes

Database mounted.

Database opened.

NEW_PRIMARY> select DATABASE_ROLE, SWITCHOVER_STATUS, GUARD_STATUS from v$DATABASE;

DATABASE_ROLE SWITCHOVER_STATUS GUARD_S

---------------- -------------------- -------

PRIMARY NOT ALLOWED NONE




Section C: Advanced Protect using EMC RecoverPoint

Overview

Introduction EMC RecoverPoint with the CLARiiON (CX) splitter provides several advantages:

• You do not need to install a dedicated splitter driver on the database servers. • You do not need to install a special hardware driver into an FCP switch. This

provides savings on both cost and manageability. • All of the splitter driver functionality is incorporated into the array.

Supporting documentation

The following documents, located on Powerlink.com, provide additional, relevant information. Access to these documents is based on your login credentials. If you do not have access, contact your EMC representative: • EMC RecoverPoint Administrator’s Guide • EMC RecoverPoint Installation Guide

Contents This section contains the following topics:

Topic See Page

EMC RecoverPoint 128

CLARiiON (CX) splitters 128

Scope 129

Best practices 129

Conclusion 130




EMC RecoverPoint

RecoverPoint EMC RecoverPoint provides integrated continuous remote replication (CRR) and

continuous data protection (CDP). The RecoverPoint family enables data to be replicated and recovered from either the local site (CDP) or at a remote site (CRR). RecoverPoint CRR complements the existing portfolio of EMC remote-replication products by adding heterogeneous replication (with bandwidth compression) in asynchronous-replication environments, which lowers multi-year total cost of ownership. RecoverPoint CDP offered as a stand-alone solution, or combined with CRR, enables you to roll back to any point in time for effective local recovery from events such as database corruption. Release 3.0 and later of RecoverPoint includes support for the CLARiiON (CX) splitter driver. These are array-based splitters that can be directly installed on a CLARiiON.

Replication with RecoverPoint

In normal replication, the target-site volumes are not accessible from the host. This is to prevent the RecoverPoint appliances and the host servers from trying to write to the volumes at the same time. Target-side processing allows the user to roll back the replication volume to a point-in time snapshot. Once the target replication volumes have been paused on this snapshot, the target host server is given access to these volumes. While the target replication volumes are paused on this snapshot, replication continues from the source site. Snapshots are still stored in the snapshot portion of the target journal volume, but are not distributed to the replication volume because it is paused on a snapshot. Once the replication is complete, the target image can be accessed by selecting Image Access from the drop-down menu.

CLARiiON (CX) splitters

CX splitters The CX splitter is integrated with each storage processor of the CLARiiON array.

This sends one copy of the I/O to the storage array and another copy to the RecoverPoint appliance. The main advantages of the CX array splitters are that they: • Reduce the cost associated with the additional host-based splitter agents or

specialized fabric. • Provide concurrent local and remote (CLR) protection. CLR replication can be

performed on the same LUNs. When an application writes to storage, the CX splitter splits the data and sends a copy of the data over Fibre Channel to the source-site RPA, and the other copy to the storage array. These requests are acknowledged by the source RPAs and are




then replicated to the remote RPAs over the IP network. The remote RPAs will write this data over to journal volumes at the remote site and the consistent data is distributed to the remote volumes later.

Scope


Only functional validation was done for the Advanced Protect solution using EMC RecoverPoint. No tuning or performance measurements were carried out. EMC plans to carry out performance testing during the next test cycle.

Best practices

Journal volumes

If there are multiple consistency groups, then you need to configure journal volumes for each consistency group on different RAID groups so that the journal of one group will not slow down the other groups. You should configure journal volumes on separate RAID groups from the user volumes. Journal volumes can be corrupted if any host writes to it other than the RecoverPoint Appliance (RPA). So, you must ensure that the journal volumes are zoned only with RPAs. RecoverPoint performs striping on journal volumes; using a large number from different RAID groups increases the performance. All journal volume LUNs should be of the same size because RecoverPoint uses the smallest LUN size, and it stripes the snapshot across all LUNs. It will not be able to stripe evenly across different sized LUNs. The size of the journal volumes should be at least 20 percent larger than the data being replicated. Journal volumes are required on both local and remote sides for Continuous Remote Replication (CRR) to support failover.

Repository volumes

Repository volumes should be at least 4 GB with an additional 2 GB per consistency group. In a CRR configuration, there must be one repository volume for the RPA cluster on the local and remote sites.

WAN compression guidelines

If you set a strong compression, this will cause CPU congestion. Conversely, if you set it to “low”, it will cause high loads. EMC recommends a 5x-10x compression ratio for Oracle databases.

Clusters The RecoverPoint clustering does not support one-to-many RPAs between sites.

The configuration should have two RPAs on both sites.




Zoning When discovering the CX array splitters, you must ensure that all the RPA ports and

all the CX SPA/SPB ports are included in the same zone. You must ensure that this zone is present on both sites. This can be accomplished by using a zone that spans multiple FCP switches, if required.

Storage volumes

The following types of storage volumes are required for RecoverPoint configuration: • Repository volume

This volume holds the configuration and marking information during the replication. At least one repository volume is required per site and this should be accessible from all the RPAs at the site.

• Journal volume This volume is used to store all the modifications. The application-specific bookmarks and timestamp details are written to the journal volume. There should be at least one journal copy per consistency group. Best practices information on the configuration of journal volumes is available in the EMC RecoverPoint Installation Guide.

• Replication set The association created between the source volume and target volume is called the replication set. The source volume cannot be greater in size than the target volume.

• Consistency group The logical group of replication sets identified for replication is called a consistency group. Consistency groups ensure that the updates to the associated volumes are always consistent and that they can be used to restore the database at any point of time.

Conclusion

Conclusion The primary benefits of deploying an Advanced Protect solution component using

RecoverPoint with CLARiiON splitters are: • Up to 148 percent total cost savings and up to 238 percent in bandwidth cost

savings • The only replication product that supports both local and remote replication • Local and remote replication with “any point in time” recovery to meet RPO/RTO

requirements • Network-based architecture optimized for availability and performance • Transparent support for heterogeneous server and storage platforms Deploying the CLARiiON splitter driver results in significant cost reductions for customers because low-cost FCP switches, such as QLogic switches, can be used in the place of high-cost intelligent switches.

Chapter 11: Test/Dev Solution Using EMC SnapView Clone




Overview

Introduction There is strong interest in configuring a writeable test/dev copy solution component

that does not impact the production server in terms of downtime or performance.


Topic See Page

CLARiiON SnapView clone 132

Best practices 132

Mount and recovery of a target clone database using Replication Manager

133

Database cloning 136




CLARiiON SnapView clone

SnapView clones

You can use EMC CLARiiON SnapView clone with consistency technology to create a test/dev solution using a restartable copy of a running Oracle RAC 11g/10g production database. Doing so has minimal impact on the production database. You can either use SnapView snapshot or SnapView clone to create the copy. • With SnapView snapshot, the I/Os that are performed on the database copy will be

handled by the same physical hardware as the production database. This can have an impact on the production database performance.

• With SnapView clone, the cloned LUNs can be created on a completely different set of RAID groups from the original LUNs. As a result, I/O performed on the database copy is handled by different physical hardware from the production database. The database copy’s I/O does not impact the production database in any way.

The SnapView clone approach requires a full copy of the production database, whereas, the SnapView snapshot approach does not. The read I/O to the production LUNs that is required to create the clone LUNs does impact the production database. However, the ongoing operation of the copy database then has no appreciable impact on the production database. We assume that the SnapView clone approach is preferable for most customers.

Use of EMC SnapView clone

A clone is a full binary copy of a source LUN. Each clone LUN must be the same size as the source LUN.

Best practices

Mounting LUNs EMC recommends using a second set of database servers to mount the LUNs and

manage them within the oracleasm kernel module. ASM writes a unique signature to each LUN and will not open a LUN containing the same signature as an existing LUN.

Consistency technology

Consistency technology allows the customer to create a copy (either virtual or physical) of a set of LUNs on either a single array or multiple arrays with the writes to all of the LUNs being in perfect order. The state of the copy LUNs is identical to the state in which the production LUNs would be if the database server was powered off. This allows you to create a restartable database copy; the database engine will perform crash recovery on this copy using the online redo logs in exactly the same manner as a power loss. Because of this unique functionality, the use of backup mode or RMAN cloning is not required, which is advantageous as both of these approaches use host and array resources and can have an impact on production database performance.




Restartable copy versus backup

A restartable copy should not be considered a substitute for a backup. Database restart following database server crash is not guaranteed by Oracle. Therefore, a crash-consistent image is not a reliable backup. Recovery cannot be performed on a crash-consistent image. The image can be restarted in the state it was in at the time the copy occurred. It cannot be recovered to the state of a later time. Therefore, EMC recommends that you use normal backup procedures to back up the production Oracle database, using either SnapView snapshot or RMAN. This does not mean, however, that a restartable copy is useless; it is very useful. For example, a restartable copy can be used for testing and development purposes. Since the creation of the restartable copy has low impact on the production database, it can be created fairly often, typically daily.

Mount and recovery of a target clone database using Replication Manager

Target clone database

We cloned a four-node RAC production database and then mounted the cloned database on a VMware host as a single-instance database. The cloning and mounting of the cloned database on the target VM host was performed using Replication Manager (RM).

Creating and executing an RM job for SnapView clone

The following table describes the main steps that are followed to create and execute an RM job for SnapView clone:

Step Action

1 In Replication Manager, select Jobs, then right-click and select New Job.

2 On the Welcome screen of the Job Wizard, choose the application set that you want to replicate, then click Next.

3 On the Job Name and Settings screen, type the job name and select: • Replication source • Replication technology name • Number of replicas to be created Click Next.

4 On the Replication Storage screen, select the storage pool that you want to use for the replica, then click Next.

5 On the Mount Options screen, select the mount options, then click Next. Note You can select the mount options at a later point in time, if required.




6 On the Starting the Job screen, choose how you want to start the job, then click Next.

7 On the Users to be Notified screen, type the email addresses of the users to be notified when a job is completed, then click Next.

8 On the Completing the Job wizard screen, save the job, then click Finish.

9 In Replication Manager, under Jobs, verify that the job has been created without any errors. If errors occur during the creation of the job, this will be indicated in the Latest Status column.

10 After the clone job has been created, select the job, then right-click and select Run to execute the job. The complete logs will be displayed during the execution.

11 In Replication Manager, under Jobs, check the Status column to verify that the job executed successfully.

Mount and recovery of a target clone database using RM

Replication Manager generates its own init.ora file as part of the replication process. This file is then placed in the directory specified by the ERM_TEMP_BASE variable (/tmp by default). This init.ora file is usually sufficient to start the database, but it does not necessarily contain all the parameters from the original init.ora file. The procedure to customize the init.ora file that is generated by Replication Manager is described in the following table.

Step Action

1 On the mount host, change to the RM client install directory:

[root@mteoraesx1-vm5 ~]# cd /opt/emc/rm/client/bin/

2 Create a new directory using the SID name for the mount, then change to that directory:

[root@mteoraesx1-vm5 bin]# mkdir mterac211

[root@mteoraesx1-vm5 bin]# cd mterac211/

3 Create a new init<SID>.ora file using the same SID name that is specified above:

[root@mteoraesx1-vm5 mterac211]# vi initmterac211.ora

4 Customize the parameter file as required. Important These parameters will be appended to the init.ora file that was generated by Replication Manager, so they must follow the correct Oracle syntax. For more information on parameter file customization, refer to EMC Solutions for Oracle Database 10g/11g for Midsize Enterprises EMC Celerra Unified Storage Platform – Best Practices Planning.




5 In Replication Manager, select the clone replication job, then right-click and select Mount.

6 Specify the correct SID name. The new init<SID>.ora file will be picked up dynamically. Those parameters will be used to start the database on the mount host in conjunction with the parameter file generated by Replication Manager.

7 On the Mount Wizard screen, select the replica to be mounted, then click Next.

8 On the Mount Options screen: a) Under Path options, select Original path.

This ensures that the database can be mounted using the same path on the target host as well.

b) Under Oracle, select Recover the database. This ensures that the target clone database will be recovered automatically after mounting by Replication Manager.

c) Click Finish.

9 In Replication Manager, verify that the mount/recovery of the clone database is started without any errors.

10 Once the job is completed successfully, verify that the clone database is open in read/write mode:

SQL> select name,open_mode from v$database;

NAME OPEN_MODE

MTERAC21 READ WRITE Replication Manager can be used to automate the complete process of cloning the

LUNs, mounting the cloned LUNs on the target host, and restoring and recovering the clone database.




Database cloning

The importance of cloning

The ability to clone a running production Oracle database is a key requirement for many customers. The creation of test and development databases, enabling of datamart and data warehouse staging, and Oracle and OS version migration are just a few applications of this important functionality.

Cloning methods

Two methods can be used for database cloning: Full clone Full clone involves taking a full copy of the entire database. Full clone is recommended for small databases or for a one-time cloning process. Incremental cloning Incremental cloning is more complex but allows you to create a clone, making a full copy on the first iteration and, thereafter, making an incremental clone for all other iterations, by copying only the changed data in order to update the clone. Incremental cloning is recommended for larger databases and for situations where there is an ongoing or continuous need to clone the production database.

Chapter 12: Migration




Overview

Migration from FCP/ASM to NFS

The ability to migrate an Oracle database across storage protocols is a frequent customer request. The EMC Oracle CSV group has tested and validated a solution component for migrating an online production Oracle database, which is mounted over FCP/ASM to a target database mounted using NFS. This is performed with minimal performance impact on the production database and no downtime

Migrating an online Oracle database

To see the steps that were followed to perform the migration operation, see Chapter 7: Testing and Validation > Section I: Migration solution component > Test procedure.




Migration diagram

The following diagram is a high-level view of the migration component.

Chapter 13: Conclusion



Chapter 13: Conclusion

Overview

Introduction The EMC Celerra unified storage platform’s high-availability features combined with

EMC’s proven storage technologies provide a high-performance, high-flexibility, and low-risk storage system for the Oracle RAC 11g/10g over FCP and NFS.

Conclusion This solution provides the following benefits:

• Reduced total cost of ownership − This solution reduces the load on the database server CPU by using EMC

Replication Manager and EMC SnapView to carry out a physical backup of an Oracle 11g or 10g production database while offloading all performance impacts of the backup operation from the production server.

• Highly flexible, easy-to-manage storage architecture. This single storage platform improves the flexibility and ease of management of the storage infrastructure with multiple protocol support and storage consolidation. By supporting multiple protocols, the Celerra unified storage platform remains an integral piece of the storage infrastructure as business needs evolve and the data center's protocol requirements change. Consolidating multiple storage devices into this single platform facilitates management by centralizing management activities and providing additional management tools.

• Reduce the complexity of backup and recovery − EMC Replication Manager eliminates the requirement for the customer to write

scripts or to manually perform replication tasks. These tasks can be fully automated and managed by Replication Manager.

• Improve data protection − Advanced Protect using RecoverPoint through the CX splitter driver provides

cost savings and superior data integrity, no disruptions during replication of data, simultaneous local and remote replication, and minimal performance impact.

− Advanced Protect using MirrorView/A through iSCSI provides simplified deployment and the use of an IP network protocol, which reduces cost and complexity.

• Eliminate the performance impact of utility operations on production databases − The ability to deploy a writeable copy of the production database provides this

deployment ability with minimal, if any, performance impact on the production database server and no downtime.

• Robust performance and scaling Running the Oracle RAC 11g/10g with Celerra unified storage platform provides the best availability, scalability, manageability, and performance for your database applications.

emc business continuity for oracle database 11g/10g · best practices ... emc business continuity...

Documents