emc vspex for virtualized oracle database 12c oltp enabled by
TRANSCRIPT
IMPLEMENTATION GUIDE
EMC VSPEX FOR VIRTUALIZED ORACLE DATABASE 12c OLTP Enabled by EMC VNXe and EMC Data Protection
VMware vSphere 5.5 Red Hat Enterprise Linux 6.4
EMC VSPEX
Abstract
This Implementation Guide describes the high-level steps and best practices required to implement the EMC® VSPEX® Proven Infrastructure for Virtualized Oracle Database 12c on a VSPEX Private Cloud with VMware vSphere enabled by EMC VNXe® and EMC Data Protection. Usage guidelines and major customer benefits are also included.
February 2015
2 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Copyright © 2015 EMC Corporation. All rights reserved. Published in the USA.
Published February 2015
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners.
For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.
EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Part Number H12969.2
Contents
3 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Contents
Chapter 1 Introduction 7
Purpose of this guide ....................................................................................... 8
Business value ................................................................................................. 8
Scope ............................................................................................................... 9
Audience .......................................................................................................... 9
Terminology ................................................................................................... 10
Chapter 2 Before You Start 11
Overview ........................................................................................................ 12
Pre-deployment tasks .................................................................................... 12
Deployment workflow .................................................................................... 13
Deployment prerequisites .............................................................................. 14
Planning and sizing the Oracle Database 12c environment ............................ 15
Overview .................................................................................................... 15
Storage pools ............................................................................................ 15
Example: Small-sized Oracle Database 12c server .................................... 15
Essential Reading .......................................................................................... 17
VSPEX Proven Infrastructure ...................................................................... 17
EMC Data Protection for VSPEX Guide ........................................................ 17
VSPEX Design and Implementation Guides ................................................ 17
Chapter 3 Solution Overview 19
Overview ........................................................................................................ 20
EMC VSPEX Proven Infrastructure ................................................................... 20
Solution architecture...................................................................................... 21
Summary of key components ......................................................................... 23
Chapter 4 Solution Implementation 25
Overview ........................................................................................................ 26
Preparing the physical components ............................................................... 26
Implementing the network ............................................................................. 26
Implementing the storage array ..................................................................... 27
Overview .................................................................................................... 27
Tables
4 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Set up initial VNXe configuration ............................................................... 27
Configure storage layout for Oracle example ............................................. 27
Provision storage for NFS example ............................................................. 29
Configure FAST VP on VNXe ........................................................................ 29
Configure FAST Cache on VNXe .................................................................. 30
Storage layout example ............................................................................. 31
Implementing ESXi and vCenter ..................................................................... 32
Preparing the Oracle Database 12c virtual machine template ........................ 32
Complete Oracle install prerequisites ........................................................ 34
Install Oracle Database 12c binaries and patches ..................................... 34
Deploying the Oracle Database 12c virtual machine ...................................... 36
Overview .................................................................................................... 36
Configure database file layout for both ASM and dNFS .............................. 36
Configure initialization parameters ............................................................ 37
Enable the HugePages setting ................................................................... 38
Create the database using DBCA ............................................................... 38
Perform database post creation activities .................................................. 39
Implement EMC Data Protection ................................................................ 40
Chapter 5 Solution Verification 41
Verifying hardware ......................................................................................... 42
Verify ESXi functionality ............................................................................. 42
Verify solution component redundancy ..................................................... 42
Use health monitoring tools ...................................................................... 42
Verifying Oracle 12c OLTP database performance .......................................... 43
Overview .................................................................................................... 43
Verify baseline performance ...................................................................... 43
Verify FAST Cache performance acceleration ............................................. 44
Chapter 6 Reference Documentation 47
Reference documentation .............................................................................. 48
EMC ........................................................................................................... 48
Oracle ........................................................................................................ 48
VMware ..................................................................................................... 48
Links .............................................................................................................. 49
Appendix A Configuration Worksheet 51
Configuration worksheet for VSPEX ................................................................ 52
Contents
5 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Figures
Figure 1. VSPEX Proven Infrastructure ....................................................... 20
Figure 2. Architecture of the validated infrastructure ................................ 22
Figure 3. Provisioning storage for VMFS example ...................................... 28
Figure 4. Provisioning storage for NFS example ......................................... 29
Figure 5. FAST VP-enabled storage pool example ...................................... 30
Figure 6. Storage Pool properties: FAST Cache option ............................... 31
Figure 7. Example of virtualized Oracle storage layout for the VNXe series 31
Figure 8. Oracle Database Configuration Assistant setup screen .............. 39
Figure 9. Performance comparison before and after FAST Cache enabled . 46
Tables
Table 1. Terminology ................................................................................ 10
Table 2. Tasks for pre-deployment ........................................................... 12
Table 3. VSPEX for virtualized Oracle deployment process ....................... 13
Table 4. Deployment prerequisites checklist............................................ 14
Table 5. Storage pools ............................................................................. 15
Table 6. Example of customer evaluation: Large-size Oracle Database 12c server ......................................................................................... 15
Table 7. Example of required resources: Large-sized user database ........ 16
Table 8. Example of additional storage layout recommendations: Large-sized Oracle Database 12c server ............................................... 16
Table 9. Solution components ................................................................. 23
Table 10. Setup tasks for physical components ......................................... 26
Table 11. Tasks for switch and network configuration ................................ 26
Table 12. Tasks for storage configuration................................................... 27
Table 13. Storage layout example on VNXe ................................................ 27
Table 14. Steps for installing and configuring ESXi, vCenter, and infrastructure servers .................................................................. 32
Table 15. Example of a virtual machine template specification .................. 33
Table 16. Oracle install prerequisites ......................................................... 34
Table 17. Installing the Oracle Database binaries and patches .................. 34
Table 18. Location of Oracle files for ASM .................................................. 36
Table 19. Location of Oracle files for NFS ................................................... 37
Tables
6 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Table 20. Tasks to verify the VSPEX installation ......................................... 42
Table 21. Tools for monitoring the solution ................................................ 43
Table 22. Example of key metrics for an Oracle Database 12c deployment 43
Table 23. Oracle Database 12c virtual machine profile .............................. 44
Table 24. Oracle Database 12c storage pool design example .................... 44
Table 25. Key performance metrics for Oracle Database 12c without FAST Cache ......................................................................................... 44
Table 26. Key metrics for Oracle Database 12c with FAST Cache ................ 45
Table 27. Comparison before and after FAST Cache enabled...................... 46
Table 28. Common server information ....................................................... 52
Table 29. ESXi server information .............................................................. 52
Table 30. Array information ........................................................................ 53
Table 31. Network infrastructure information ............................................. 53
Table 32. VLAN information ........................................................................ 53
Table 33. Service accounts ........................................................................ 54
Chapter 1: Introduction
7 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Chapter 1 Introduction
This chapter presents the following topics:
Purpose of this guide .................................................................................... 8
Business value ............................................................................................. 8
Scope ........................................................................................................... 9
Audience ...................................................................................................... 9
Terminology ............................................................................................... 10
Chapter 1: Introduction
8 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Purpose of this guide
EMC® VSPEX® Proven Infrastructures are optimized for virtualizing business-critical applications. VSPEX provides modular solutions built with technologies that enable faster deployment, more simplicity, greater choice, higher efficiency, and lower risk.
VSPEX provides partners with the ability to design and implement the virtual assets required to support a fully integrated virtualized solution for Oracle relational database management systems on a VSPEX private cloud infrastructure.
The VSPEX for virtualized Oracle Database 12c infrastructure provides customers with a system capable of hosting a virtualized database solution that is scalable and delivers a constant performance level. This solution uses VMware vSphere 5.5 to provide virtualization supported by the EMC VNXe® storage array. The compute and network components, while vendor-definable, are designed to provide redundancy and sufficient power to handle the processing and data needs of the virtual machine environment.
This Implementation Guide describes how to best implement the VSPEX Proven Infrastructure for virtualized Oracle Database 12c on a VSPEX Private Cloud with VMware vSphere.
Business value
This VSPEX Proven Infrastructure is focused on helping EMC partners understand the value that the VNXe series and EMC backup and recovery systems bring to customers with Oracle Database 12c deployments who often have growing, isolated IT environments running server-centric applications and who face increasing Oracle backup and recovery issues.
This VSPEX solution is designed to meet your Oracle database challenges while enabling customers to increase performance, scalability, reliability, and automation. By consolidating their database applications on VNXe, they can consolidate onto a single centralized storage platform that enables them to more effectively manage the exploding growth in data that is challenging businesses today. This solution has been sized and proven to do the following:
Deploy your systems faster, saving time and effort with EMC Proven Solutions
Increase performance and scalability out of the box
Minimize storage requirements and reduce costs
Chapter 1: Introduction
9 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Scope
This Implementation Guide describes the high-level steps required to deploy the VSPEX solution for Oracle Database 12c on a VSPEX Private Cloud for VMware vSphere with an EMC VNXe3200™ storage array. This Implementation Guide assumes that a VSPEX Proven Infrastructure already exists in the customer’s environment.
Audience
This guide is intended for internal EMC personnel and qualified EMC VSPEX partners. This guide assumes that the VSPEX partners who intend to deploy this solution are:
Qualified by EMC to sell, install, and configure the EMC VNX family of storage systems
Qualified to sell, install, and configure the network and server products required for VSPEX Proven Infrastructures
Certified for selling VSPEX Proven Infrastructures
Partners implementing this solution should also have the necessary technical training and background to install and configure:
VMware vSphere virtualization platforms 5.5
Red Hat Enterprise Linux 6.4
Oracle Database 12c
This document provides external references where applicable. EMC recommends that partners implementing this solution are familiar with these documents. For details, refer to Essential reading and Chapter 6: Reference documentation.
Chapter 1: Introduction
10 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Terminology
Table 1 lists the terminology used in this guide.
Table 1. Terminology
Term Definition
AWR Automatic Workload Repository is a monitoring utility bundled with Oracle Database 10g and later releases
dNFS Direct Network File System client can be used to provide access to Network File System (NFS) storage devices
OLTP Online transaction processing is a class of systems that manage applications for data entry and retrieval transaction processing
SGA System global area is a group of shared memory structures that contain data and control information from one oracle database instance
PGA Program Global Area is memory specific to an operating process or thread that is not shared by other processes or threads on the system
VMDK VMware Virtual Machine Disk is an open file format that is a container for virtual hard disk drives for virtual machines
VMFS VMware Virtual Machine File System is a cluster file system that uses storage virtualization for multiple installations of VMware ESX Server
Chapter 2: Before You Start
11 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Chapter 2 Before You Start
This chapter presents the following topics:
Overview .................................................................................................... 12
Pre-deployment tasks................................................................................. 12
Deployment workflow ................................................................................. 13
Deployment prerequisites .......................................................................... 14
Planning and sizing the Oracle Database 12c environment.......................... 15
Essential Reading ....................................................................................... 17
Chapter 2: Before You Start
12 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Overview
This chapter provides an overview of important information you need to be aware of, documents you need to be familiar with, and tasks you need to perform before you start implementing your VSPEX for virtualized Oracle Database 12c solution.
The EMC VSPEX for Virtualized Oracle Database 12c OLTP Design Guide for this solution describes how to size and design your solution and how to select the right VSPEX Proven Infrastructure on which to layer Oracle Server. The deployment examples in this Implementation Guide are based on the recommendations and examples in the Design Guide.
EMC recommends that you complete the pre-deployment tasks as described in the following section before you deploy Oracle Server on a VSPEX Proven Infrastructure.
Pre-deployment tasks
Pre-deployment tasks include those not directly related to the environment installation and configuration, but are prerequisites for installation. These tasks include collecting hostnames, IP addresses, license keys, and installation media. Perform the tasks in Table 2 to decrease the time required at the customer site.
Table 2. Tasks for pre-deployment
Task Description Reference
Gather documents Get the related documents listed in Essential reading. These are used throughout this document to provide details on setup procedures and deployment best practices for the various components of the solution.
Essential reading
Gather tools Get the required and optional tools for the deployment. Use Table 4 to confirm that all equipment, software, and appropriate licenses are available before the deployment process.
Deployment prerequisites
Gather data Get the customer-specific configuration data for networking, naming, and required accounts. Complete the Customer Oracle Database 12c configuration worksheet for reference during the deployment process.
Appendix A Configuration Worksheet
Chapter 2: Before You Start
13 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Deployment workflow
EMC recommends that you follow the process flow in Table 3 to design and implement your VSPEX Proven Infrastructure for virtualized Oracle Database 12c OLTP solution.
Table 3. VSPEX for virtualized Oracle deployment process
Step Action
1 Use the VSPEX qualification worksheet to collect user requirements. The qualification worksheet is in “Appendix A” of the companion EMC VSPEX For Virtualized Oracle Database 12c OLTP Design Guide.
2 Use the VSPEX Sizing Tool to determine the recommended VSPEX Proven Infrastructure for a virtualized Oracle Database 12c OLTP solution. For more information, refer to the VSPEX Sizing Tool on the EMC Business Value Portal.
Note: You must register the first time you access the tool. If the VSPEX Sizing Tool is not available, you can manually size the application by using the sizing guidelines in “Appendix B” of the companion Design Guide.
3 Refer to the companion Design Guide to determine the final design for the VSPEX Proven Infrastructure for virtualized Oracle Database 12c OLTP.
Note: Consider all application requirements, not just Oracle application requirements.
4 Refer to the applicable VSPEX Proven Infrastructure document in Essential reading to select and order the right solution.
5 To deploy and test your VSPEX solution, use this Implementation Guide.
Chapter 2: Before You Start
14 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Deployment prerequisites
Table 4 itemizes the hardware and software requirements to configure the solution. For additional information, refer to the hardware and software tables in the EMC VSPEX for Virtualized Oracle Database 12c OLTP Design Guide.
Table 4. Deployment prerequisites checklist
Component Description Version Reference notes
Hardware Physical servers: Sufficient physical server capacity to host the required number of virtual machines, as recommended by the companion Design Guide and VSPEX Sizing Tool.
N/A EMC VSPEX Private Cloud: VMware vSphere 5.5 for up to 200 Virtual Machines
Virtual servers: VMware vSphere servers that host the virtual infrastructure servers. This requirement may be covered in the existing infrastructure.
vSphere 5.5
Networking: Switch port capacity and capabilities as required by the virtual server infrastructure.
N/A
EMC VNXe array: Multiprotocol storage array with the required disk layout.
Note: The arrays must provide sufficient additional storage for applications.
VNXe3200
Software EMC VNXe Operating Environment ( OE) 3.0.1 EMC Online Support
EMC Unisphere for VNXe 3.0.1
EMC vStorage APIs for Array Integration Plug-in Release for vSphere 5.5
VMware ESXi 5.5 www.vmware.com
VMware vCenter Server 5.5
Oracle Database Server 12.1.0 www.oracle.com
Red Hat Enterprise Linux 6.4 www.redhat.com
Licenses VMware vSphere license keys. 5.5 www.vmware.com
Oracle Database Server license key 12c Release 1 Standard or Enterprise Edition
www.oracle.com
Chapter 2: Before You Start
15 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Planning and sizing the Oracle Database 12c environment
Follow the recommendations and VSPEX Sizing Tool proposals introduced in the Design Guide to plan and size your Oracle Database 12c environment on the VSPEX Proven Infrastructure.
We used the storage pool configuration as detailed in Table 5 in this VSPEX for virtualized Oracle Database 12c solution.
Table 5. Storage pools
Pool name Purpose
VSPEX private cloud pool Store the virtual machine OS files and database binary files
Oracle Database data pool Store database datafiles
Oracle Database redo pool Store database redo log files
Oracle Database FRA pool (optional)
Store database backup files
Note: Fast Recovery Area (FRA) is a feature Oracle recommends that you enable to store all backup files in one place. If the FRA feature is enabled, the sizing logic for Oracle archived log and backup files is different from that for Oracle datafiles and redo log files. For detailed support, consult your EMC representative.
Complete the evaluation as shown in Table 6. A similar example is described in the companion Design Guide.
Table 6. Example of customer evaluation: Large-size Oracle Database 12c server
Question Answer example
Do you have an existing Oracle Database 12c server in the environment for which you would like to size?
Yes
How many databases do you want to deploy? 1
What is the size of the user database (GB)? 1,000
What is the annual growth rate? 10%
Do you intend to use FAST Cache? No
Do you intend to use FAST VP? No
What is the maximum number of IOPS (read/write)?
2,900/1,900
Overview
Storage pools
Example: Small-sized Oracle Database 12c server
Chapter 2: Before You Start
16 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Question Answer example
(Optional) What is the expected number of concurrent users at peak loads?
4,500
You will see results similar to those shown in the following tables after you have received a completed qualification worksheet from the customer and entered the answers into the VSPEX Sizing Tool:
Table 7 lists the number of virtual machines and their characteristics.
Table 8 lists the additional storage hardware that is required to run the OLTP workload. This is in addition to the VSPEX private cloud pool.
Table 7 and Table 8 are examples based on the customer information provided in Table 6. The suggested storage layout is in addition to the VSPEX private cloud pool.
Table 7. Example of required resources: Large-sized user database
vCPU Memory OS volume capacity
OS volume IOPS No. of virtual machines
32 98 GB 100 GB 100 1
Table 8. Example of additional storage layout recommendations: Large-sized Oracle
Database 12c server
Pool name RAID type Disk type Disk capacity No. of disks
VSPEX private cloud pool RAID 5 SAS disks 10,000 rpm 600 GB 5
Oracle database data file pool RAID 5 SAS disks 10,000 rpm 600 GB 30
Oracle database log file pool RAID 1/0 SAS disks 10,000 rpm 600 GB 44
Note: To plan and size your Oracle Database 12c deployment on the VSPEX infrastructure, follow the recommendations in the companion Design Guide.
Chapter 2: Before You Start
17 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Essential reading
Before implementing the solution described in this document, EMC recommends that you read the following documents, available from the VSPEX space on the EMC Community Network or from the VSPEX Proven Infrastructure Pages on EMC.com. If you do not have access to a document, contact your EMC representative.
Refer to the following EMC VSPEX Proven Infrastructure Guide:
EMC VSPEX Private Cloud VMware vSphere 5.5 for up to 200 virtual machines Enabled by Microsoft Windows Server 2012 R2, EMC VNXe3200, and EMC Powered Backup
Refer to the following Design and Implementation Guide for EMC Data Protection for VSPEX:
EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11gR2 Design and Implementation Guide
Refer to the following VSPEX documents:
EMC VSPEX for Virtualized Oracle Database 12c OLTP Design Guide
VSPEX Infrastructure as a Service with VMware vCloud Suite Design and Implementation Guide
VSPEX Proven Infrastructure
EMC Data Protection for VSPEX Guide
VSPEX Design and Implementation Guides
Chapter 2: Before You Start
18 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Chapter 3: Solution Overview
19 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Chapter 3 Solution Overview
This chapter presents the following topics:
Overview .................................................................................................... 20
EMC VSPEX Proven Infrastructure ............................................................... 20
Solution architecture .................................................................................. 21
Summary of key components ...................................................................... 23
Chapter 3: Solution Overview
20 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Overview
This chapter provides an overview of the VSPEX Proven Infrastructure for Oracle Database 12c and the key technologies used in this solution. The solution described in this Implementation Guide includes servers, storage, network components, and Oracle Database 12c components.
The solution enables customers to quickly and consistently deploy a virtualized Oracle Database 12c in the VSPEX Proven Infrastructure.
EMC VSPEX Proven Infrastructure
A VSPEX Proven Infrastructure, as shown in Figure 1, is a modular virtualized infrastructure validated by EMC and delivered by EMC VSPEX partners. VSPEX includes virtualization, server, and network layers, and EMC storage and backup, designed by EMC to deliver reliable and predictable performance.
Figure 1. VSPEX Proven Infrastructure
Chapter 3: Solution Overview
21 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
VSPEX provides the flexibility to choose network, server, and virtualization technologies to create a complete virtualization solution. VSPEX delivers faster deployment for EMC partner customers, with greater simplicity, efficiency, and choice, and lower business risk.
You can deploy application-based solutions such as Oracle Server on VSPEX Proven Infrastructures. We validated the VSPEX Proven Infrastructure for virtualized Oracle Server solution using VNXe and a VMware virtualized Linux Server platform to provide storage and server hardware consolidation. You can centrally manage the virtualized infrastructure, which enables you to efficiently deploy and manage a scalable number of virtual machines and associated shared storage.
Solution architecture
Figure 2 shows the architecture that characterizes the infrastructure validated for an Oracle Database 12c overlay on a VSPEX infrastructure.
We used the VSPEX Sizing Tool for Oracle Database 12c to determine the amount of compute resources for each Oracle database. Figure 2 shows the validated architecture for the VSPEX Proven Infrastructure for Oracle 12c Server.
We used the VSPEX Sizing Tool to determine the recommended storage layout for Oracle Database 12c and the virtual infrastructure pool in the VNXe3200 storage array. The sizing tool is located on the EMC Business Value Portal.
Note: The minimum Oracle version for this solution is 12.1.0.1. We refer to this as 12c throughout this document.
Chapter 3: Solution Overview
22 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Figure 2. Architecture of the validated infrastructure
Chapter 3: Solution Overview
23 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Summary of key components
Table 9 summarizes the key components used in this solution. The companion Design Guide provides more details about the individual components.
Table 9. Solution components
Layer Components
Application Oracle Database 12c
Virtualization VMware vSphere 5.5
Compute Minimum amount of compute layer resources required, as defined by VSPEX
Note: You can implement any server hardware that meets these requirements.
Network Minimum number of network ports required, as defined by VSPEX
Note: VSPEX defines the minimum number of network ports required and provides general guidance on network architecture, and enables you to implement any network hardware that meets these requirements.
Storage EMC VNXe3200
OS Red Hat Enterprise Linux 6.4
Backup EMC Data Protection solutions
Chapter 3: Solution Overview
24 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Chapter 4: Solution Implementation
25 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Chapter 4 Solution Implementation
This chapter presents the following topics:
Overview .................................................................................................... 26
Preparing the physical components ............................................................ 26
Implementing the network .......................................................................... 26
Implementing the storage array .................................................................. 27
Implementing ESXi and vCenter .................................................................. 32
Preparing the Oracle Database 12c virtual machine template ...................... 32
Deploying the Oracle Database 12c virtual machine .................................... 36
Chapter 4: Solution Implementation
26 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Overview
This chapter describes the steps required to install, configure, and implement an Oracle Database 12c solution on the VSPEX Proven Infrastructure environment enabled by VMware vSphere 5.5 and VNXe.
Note: If you already have a VSPEX Proven Infrastructure environment, you can disregard any implementation steps that you have already completed.
Preparing the physical components
This section lists the tasks needed to prepare the solution’s physical components. After you complete the tasks described in Table 10, the new hardware components will be racked, cabled, powered on, and ready for network connections.
Table 10. Setup tasks for physical components
Task Description Reference
Preparing network switches
Install the switches in the rack and connect them to power.
Vendor documentation
Preparing servers Install the servers in the rack and connect them to power.
Vendor documentation
Preparing VNXe Install the VNXe in the rack and connect it to power.
EMC VNXe3200 Installation Guide
Implementing the network
This section describes requirements for the network infrastructure that is needed to support the solution architecture. Table 11 provides a summary of the tasks for switch and network configuration along with references for further information.
Table 11. Tasks for switch and network configuration
Task Description Reference
Configuring network
Configure storage network and host infrastructure networking as specified in the solution reference architecture.
VSPEX Proven Infrastructure Guide, listed in Essential reading
Completing network cabling
Connect the following:
Switch interconnect ports
VNXe ports
ESXi NIC/FC ports
N/A
Chapter 4: Solution Implementation
27 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Task Description Reference
Configuring VLAN
Configure private and public VLANs as required.
Vendor documentation
Configuring VMware vSphere networking
Create virtual switches for separation of application, management, and storage networks.
VMware vSphere Networking Guide
Refer to the companion Design Guide for details on the network implementation.
Implementing the storage array
This section describes how to configure the VNXe storage array. Table 12 provides a summary of the tasks for storage configuration and references for further information.
Table 12. Tasks for storage configuration
Task Description Reference
Setting up the initial VNXe configuration
Configure the IP address information and other key parameters on the VNXe array.
EMC VNXe3200 Installation Guide
VNXe Series Configuration Worksheet Provisioning
storage Provision the storage for VMFS datastore for Oracle Server data or create Network File System (NFS) that will be mounted on VMware hosts
Ensure that network interfaces, IP address information, and other key parameters, such as DNS and NTP, are configured on the VNX before provisioning the storage. Refer to the relevant VSPEX Proven Infrastructure Guide in Essential reading for more information on how to configure VNXe.
Follow the recommendations and VSPEX Sizing Tool proposals introduced in the Design Guide before you provision the storage.
Table 13 shows an example of a storage layout for Oracle on VNXe.
Table 13. Storage layout example on VNXe
Storage pool name RAID type Disk type No. of disks
Oracle Database data FAST Cache enabled pool
RAID 5 (4+1) 10,000 rpm SAS disks 30
RAID 1 (1+1) FAST Cache SSD 2
Overview
Set up initial VNXe configuration
Configure storage layout for Oracle example
Chapter 4: Solution Implementation
28 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Storage pool name RAID type Disk type No. of disks
Oracle Database redo pool
RAID 10 (2+2) 10,000 rpm SAS disks 4
Creating a pool with the appropriate number of disks
To create a pool:
1. In Unisphere, select Storage > Storage Configuration > Storage Pools.
2. Click Create to start the Storage Pool wizard. Manually create a new pool using SAS disks:
Use RAID 5 (4+1) for a storage pool with 30 drives for the database datafile pool. This is an example of the large-sized Oracle database.
In this example, for database redo log files, use RAID 10 (2+2) for a storage pool with 4 drives.
Note: Create your hot spare disks at this point. For more information, refer to the EMC VNXe3200 Installation Guide.
Follow the recommendations and VSPEX Sizing Tool proposals in the Design Guide to provision the storage for the VMFS datastore.
Figure 3. Provisioning storage for VMFS example
Chapter 4: Solution Implementation
29 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Figure 3 shows that Oracle databases use different VMDKs to contain the OS, Oracle binaries, Oracle datafiles, and redo log files. Oracle databases also use separate datastores that are mapped and dedicated to their Oracle ASM disk groups for datafiles, redo log files, and other database files.
Note: Figure 3 is only an example and actual configurations for customer deployments can differ.
Refer to the Design Guide for more information about VMFS recommendations and design.
Oracle Database 12c includes a feature for storing Oracle datafiles on a NAS device by using Oracle Direct NFS (dNFS). With dNFS over IP, all database objects are accessible through an NFS mount. Datafiles, online redo log files, and other database files are accessed with dNFS, as shown in Figure 4.
Refer to the Design Guide for more information about the storage layout recommendations and design.
Figure 4. Provisioning storage for NFS example
FAST VP increases disk resource efficiency and achieves better performance at a lower cost. FAST VP can automatically adapt to changing business needs by adding additional FAST VP SSDs to the Oracle Database datafile pool.
In VNXe, FAST VP enables moving Oracle Server data automatically between FAST VP SSDs and SAS tiers with a slice granularity of 256 MB. This helps to
Provision storage for NFS example
Configure FAST VP on VNXe
Chapter 4: Solution Implementation
30 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
reduce the response time dramatically and improve the Oracle Database OLTP performance.
Enabling FAST VP on VNXe is a transparent operation for Oracle Database Server. No reconfiguration or downtime is necessary. EMC recommends that all the LUNs in the pool have the same tiering policy and that you set the FAST policy for the participating pool LUNs to Start High and Auto-Tier for the Oracle database datafile pool.
Refer to the relevant VSPEX Proven Infrastructure Guide in Essential reading for detailed steps about how to configure FAST VP at the LUN level.
Figure 5 shows an example of the storage pools created for the VNXe system in this solution. The storage pools use 30 SAS disks and FAST VP to store Oracle datafiles.
Figure 5. FAST VP-enabled storage pool example
By using SSDs in the array as an extended read-and-write cache, FAST Cache increases IOPS and dramatically decreases database response times compared to the high cost, all SAS configuration.
Enabling FAST Cache is a transparent operation to Oracle Database Server and no reconfiguration or downtime is necessary. Refer to the Design Guide for more details.
Perform the following steps to create and configure FAST Cache:
1. Create FAST Cache on the VNXe storage array.
Refer to the relevant VSPEX Proven Infrastructure Guide listed in Essential reading for detailed steps.
2. In Unisphere, click Storage and select Storage Configuration.
Configure FAST Cache on VNXe
Chapter 4: Solution Implementation
31 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
3. Select Storage Pools, select the Oracle data pool, and then click Details.
4. Click Settings.
5. Click Allow this pool to use the FAST Cache to enable FAST Cache, as shown in Figure 6.
Figure 6. Storage Pool properties: FAST Cache option
Figure 7 shows the details of storage layout for VNXe. This is an example only. The number of disks used in the VSPEX private cloud pool and Oracle database datafile or redo log file pool may vary according to the customer’s requirements.
Refer to the Design Guide and the relevant VSPEX Proven Infrastructure Guide listed in Essential reading for more information about the storage layout recommendations and design.
Figure 7. Example of virtualized Oracle storage layout for the VNXe series
Storage layout example
Chapter 4: Solution Implementation
32 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Implementing ESXi and vCenter
This section describes the installation and configuration requirements to support the architecture, including ESXi hosts, vCenter, and infrastructure servers. Table 14 describes the tasks to complete.
Table 14. Steps for installing and configuring ESXi, vCenter, and infrastructure servers
Task Description Reference
Installing ESXi Install the ESXi 5.5 hypervisor on the physical servers that are being deployed for the solution.
vSphere Installation and Setup
Configuring and deploying vCenter Server
Install VMware vCenter Server 5.5. vCenter Server and Host Management
Installing vCenter Server 5.5 best practices
Configuring ESXi networking
Configure ESXi networking including: NIC trunking, VMkernel port, virtual machine port groups, and jumbo frames.
vSphere Networking
Creating a virtual data center
Create a virtual data center. vCenter Server and Host Management
Applying vSphere license keys
Type the vSphere license keys in the vCenter licensing menu.
vSphere Installation and Setup
Adding ESXi hosts Connect vCenter to ESXi hosts. vCenter Server and Host Management
Configuring vSphere clustering (optional)
Create a vSphere cluster. vSphere Resource Management
Performing array ESXi host discovery
Perform ESXi host discovery in the Unisphere console.
vSphere Resource Management
Refer to the relevant VSPEX Proven Infrastructure Guide in Essential reading for details on ESXi and vCenter.
Preparing the Oracle Database 12c virtual machine template
VMware provides the ability to create a template that you can use as a master copy to quickly create and provision virtual machines. By using a template, you can install a guest OS and apply it to a virtual machine with application users and software configured and ready for use with minimal administrative intervention. This minimizes deployment time and avoids repetitive installation and configuration tasks for each virtual machine that is required.
Customization specifications, maintained in vCenter, further simplify the rollout of virtual machines. A deployment wizard, automation tool, or script
Chapter 4: Solution Implementation
33 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
can use these templates to automatically create or amend server settings (such as server name, time zone, and network configuration) prior to building the new virtual machine.
This section documents the preparation of a VMware virtual machine template to enable rapid deployment of new Oracle Database 12c environments.
Complete the following steps to configure the template with the requirements and prerequisites for the Oracle software install:
1. Create a virtual machine with the following resources:
2 vCPUs
8 GB of vRAM
100 GB virtual disk
2. Install the operating system and RPM packages.
3. Configure the system and kernel parameters.
4. Create the OS groups and users.
5. Install and detach the Oracle binaries that are ready for cloning.
6. Convert the virtual machine to a VMware template.
Table 15 provides an example of a virtual machine template specification.
Table 15. Example of a virtual machine template specification
Component Description
CPU 2 vCPUs
Memory 8 GB
Virtual disk 100 GB with:
8 GB for the swap partition
92 GB for the bootable root partition
Operating system Red Hat Enterprise Linux Server release 6.4
Kernel 2.6.32
Network interfaces Eth0: public/management IP network
Eth1: Storage Network
OS users Username: oracle
OS groups Group names: oinstall and dba
Database binaries Oracle 12.1.0.1
Chapter 4: Solution Implementation
34 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Table 16 provides the steps to install and configure the operating system and Oracle prerequisites in the virtual machine before converting it to a template.
Table 16. Oracle install prerequisites
Task Description Reference (on the Oracle website)
Installing the guest OS
Install the Red Hat Enterprise Linux 6.4 64-bit operating system and configure the network interfaces.
Red Hat Enterprise Linux 6 Installation Guide
Installing VMware tools
Install VMware tools using Red Hat Package Manager (RPM)
Installing VMware Tools in a Linux virtual machine using Red Hat Package Manager (RPM) (VMware Knowledge Base Article ID 1018392)
Installing the required RPM packages
Install or update required RPM packages Oracle Database Installation Guide 12c Release 1 (12.1) for Linux
Setting the kernel parameters
Configure the kernel with Oracle-specific settings for semaphores, virtual and shared memory, network, and asynchronous I/O.
Setting the resource limits for the Oracle user
Check and adjust the resource limits for the Oracle software installation users
Creating the required software directories
Create the required directories for the Oracle software
Configuring HugePages
Configure HugePages to reduce the number of memory pages and virtual memory management overhead.
Latest version of My Oracle Support Note ID 1392497.1: USE_LARGE_PAGES To Enable HugePages
Table 17 includes the steps required to install the Oracle Database 12c binaries and patches.
Table 17. Installing the Oracle Database binaries and patches
Task Description Reference (on the Oracle website)
Installing the Oracle database binaries
Run the Oracle Universal installer and select the following installation option, which installs only the database binaries: Install database software only
Oracle Database Installation Guide 12c Release 1 (12.1) for Linux
Installing the latest PSU and CPU
Install the latest database Patch Set Update (PSU). A user ID, password, and a support contract are required.
Latest available version of Critical Patch Updates, Security Alerts, and Third Party Bulletins
Complete Oracle install prerequisites
Install Oracle Database 12c binaries and patches
Chapter 4: Solution Implementation
35 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Task Description Reference (on the Oracle website)
Enabling dNFS Enable Oracle dNFS (direct network file system) if Oracle Database is not shipped with Direct NFS enabled by default.
Oracle Database Installation Guide 12c Release 1 (12.1) for Linux
Chapter 4: Solution Implementation
36 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Deploying the Oracle Database 12c virtual machine
The following methods are available for deploying a virtual machine from a template with custom specifications:
Choosing Deploy Virtual Machine from this Template in vCenter
Choosing Customize using an existing customization specification in vCenter
Use the VSPEX for Virtualized Oracle Qualification Worksheet in the Design Guide to size the virtualization infrastructure and the overall NFS file systems used to store the Oracle database. That Design Guide also provides the Oracle initialization parameters required by the database.
Determine the size of the individual datafiles based on the schema design and table space placement of tables and indexes. Refer to Configure database file layout for both ASM and dNFS for more details.
Refer to the Design Guide and Configure initialization parameters for guidance on configuring the initialization parameters.
In this solution the Oracle 12c Database was built with two configurations to test the unified storage platform solution with different protocols. An ASM implementation with a RAID-protected ASM disk group and a dNFS implementation with a RAID-protected NFS file system were used to build the database. The back-end storage layout, excluding the file system type, is almost the same for both configurations.
Database file layout for ASM
Table 18 lists datafiles, online redo log files, CRS files, and other database files that reside on Oracle ASM-managed storage. The database was built with several distinct ASM disk groups: +DATA, +REDO, and +CRS.
Table 18. Location of Oracle files for ASM
Content Location
Database binary files Database server’s local disk, or the file system dedicated to virtual servers
Datafiles, control files +DATA
Online redo log files +REDO
OCR and voting disk files +CRS
Oracle ASM is the default file system in this solution, but Red Hat 6.4 supports multiple file systems, such as VFAT, ext2,ext3, ext4, and ReiserFS. Oracle
Overview
Configure database file layout for both ASM and dNFS
Chapter 4: Solution Implementation
37 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
generally does not certify file systems, but Linux is a specific case. The current support includes ext3, ext4, and OCFS2.
Database file layout for dNFS
Oracle Database components, such as datafiles and online redo log files, are stored by using Oracle Direct NFS (dNFS). These file systems use the appropriate RAID level and number of disks for each type of file. Table 19 lists the datafiles, online redo log files, and other database files and indicates where they reside.
Table 19. Location of Oracle files for NFS
Content Location
Database binary files Database server local disk or file system dedicated for virtualized servers
Datafiles, control files Spread across file systems dedicated to Oracle datafiles
Online redo log files Multiplexed across two different file systems dedicated to redo log files
EMC recommends using the Oracle dNFS client, which offers improved performance over the operating system-supplied for the NFS client. The Oracle dNFS client provides enhanced resiliency and performance by providing automatic failover on Gb Ethernet fabric, and performing concurrent I/O that bypasses any OS caches or write-order locks. dNFS also performs asynchronous I/O that allows processing to continue while the I/O request is submitted and processed.
Note: Oracle recommends that you set the NFS buffer size parameters to 32,768 in the system configuration file /etc/fstab. However, you can use the file oranfstab to specify additional Oracle Database specific options for Direct NFS. For more details, refer to Database Installation Guide 12c Release 1 (12.1) for Linux on the Oracle website.
Use one of the following methods to configure the initialization parameters:
Automatic Shared Memory Management
Implement Automatic Shared Memory Management (ASMM) to automate the management of shared memory structures. The following shared memory structures are automatically managed:
DB_CACHE_SIZE
SHARED_POOL_SIZE
LARGE_POOL_SIZE
JAVA_POOL_SIZE
STREAMS_POOL_SIZE
Configure initialization parameters
Chapter 4: Solution Implementation
38 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
ASMM enables the Oracle database to distribute memory between these components based on workload requirements. These components are considered auto tuned parameters. The Memory Manager (MMAN) background process coordinates the sizing of these components and moves memory to where it is needed most.
Set the following initialization parameters to implement this feature:
SGA_TARGET set to a nonzero value
STATISTICS_LEVEL set to TYPICAL (or ALL)
Automatic Memory Management
Do not use Oracle Automatic Memory Management (AMM) because AMM is incompatible with HugePages. If you want to use HugePages make sure that both MEMORY_TARGET and MEMORY_MAX_TARGET initialization parameters are not set.
For more information, refer to My Oracle Support Note ID 749851.1 on the Oracle website.
I/O operations for file system files
Set FILESYSTEMIO_OPTIONS = SETALL to enable direct I/O and asynchronous I/O for file system files.
With asynchronous I/O, normal processing continues while the I/O request is submitted and processed. This enables asynchronous I/O to bypass some of the performance bottlenecks associated with I/O operations. Direct I/O is available in the file system and delivers data directly to the application without caching in the file system buffer cache.
Direct NFS does not depend on the value of FILESYSTEMIO_OPTIONS. Direct NFS always issues asynchronous and direct I/O, as it does not depend on OS support. Set FILESYSTEMIO_OPTIONS = SETALL if the OS supports it to back up the OS NFS client in case of a misconfiguration.
HugePages was enabled in the Linux Kernel as part of the template creation process, as described in Table 16. HugePages is crucial for faster Oracle database performance on Linux if you have a large amount of RAM and SGA size. You need to configure HugePages if your combined database SGAs are more than 8 GB.
Refer to the latest version of My Oracle Support Note ID 361468.1: HugePages on Oracle Linux 64-bit on the Oracle website for more details.
The Database Configuration Assistant (DBCA) is the preferred way to create a database because it is a more automated approach and the database is ready
Enable the HugePages setting
Create the database using DBCA
Chapter 4: Solution Implementation
39 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
to use when DBCA completes. Start DBCA as a stand-alone tool to create an Oracle Database, as shown in Figure 8.
Type the following command in a UNIX or Linux shell or Windows command prompt and press Enter to start DBCA: dbca
Note: The dbca utility is typically located in the ORACLE_HOME/bin directory.
Figure 8. Oracle Database Configuration Assistant setup screen
Oracle DBCA runs in interactive mode. You can either modify one of the preconfigured database types or customize the database by using the screens in Oracle DBCA.
General Purpose/Transaction Processing is a preconfigured database template that you can create and customize during the installation. This involves amending the following:
Initialization parameters
Size and placement of the database files to use the NFS mount points
After you create a database as described in Create the database using DBCA, the instance is open and available for normal database use. However, the Oracle database does not automatically start or stop with the virtual machine.
Perform database post creation activities
Chapter 4: Solution Implementation
40 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Refer to My Oracle Support Note ID 222813.1: How to Automate Startup/Shutdown of Oracle Database on Linux on the Oracle website to automate the Oracle Database 12c startup and shutdown processes.
All VSPEX solutions are sized and tested with EMC Data Protection products, including EMC Avamar and EMC Data Domain. If your solution includes EMC Data Protection components, refer to EMC Backup and Recovery Options for VSPEX for Virtualized Oracle Database 11gR2 Design and Implementation Guide for detailed information on implementing these options into your VSPEX solution.
Implement EMC Data Protection
Chapter 6: Solution Verification
41 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Chapter 5 Solution Verification
This chapter presents the following topics:
Verifying hardware ..................................................................................... 42
Verifying Oracle 12c OLTP database performance ....................................... 43
Chapter 6: Solution Verification
42 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Verifying hardware
This section provides a list to review after configuring this VSPEX solution. Use the information in this section to verify the configuration and functionality of specific aspects of the solution, and to ensure that the configuration supports core availability requirements. Table 20 describes the required verification tasks.
Table 20. Tasks to verify the VSPEX installation
Task Reference
Verify the basic ESXi functionality of the solution with a post-installation checklist.
Using EMC VNX storage with VMware vSphere TechBook
Verify the redundancy of the following solution components:
Storage
ESXi host
Network switch
Vendor documentation
Use the monitoring tool to monitor the health of the VSPEX Proven Infrastructure for virtualized Oracle databases
VSPEX documentation
EMC recommends that you verify the ESXi configurations before deploying the system to production on each ESXi server. Refer to the relevant VSPEX Proven Infrastructure Guide in Essential reading for more detailed information.
You can test specific scenarios related to maintenance or hardware failure to ensure that the various components of the solution maintain availability requirements. EMC recommends that you verify redundancy of the solution components, including storage, ESXi hosts, and network switches.
Refer to the relevant VSPEX Proven Infrastructure Guide in Essential reading for details.
Health monitoring is a simplified measurement that reflects the reliability, stability, and performance of the entire VSPEX Proven Infrastructure for a virtualized Oracle database. Table 21 lists some tools for you to monitor and troubleshoot your VSPEX solution.
Verify ESXi functionality
Verify solution component redundancy
Use health monitoring tools
Chapter 6: Solution Verification
43 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Table 21. Tools for monitoring the solution
Tool Description
Esxtop The esxtop tool provides a real-time view (updated every five seconds, by default) of ESXi hypervisor performance metrics, such as CPU utilization. These performance metrics enable you to measure and troubleshoot Oracle database performance-related issues.
Automatic Workload Repository (AWR)
The Automatic Workload Repository (AWR) collects, processes, and maintains performance statistics for problem detection and self-tuning. This data is stored both in memory and in the database. The gathered data can be displayed in both reports and views.
EMC recommends that you use these tools to monitor the performance of the entire Oracle Database environment. Refer to the companion Design Guide for details about performance verification and test methodology.
Verifying Oracle 12c OLTP database performance
This section provides an example performance test to verify Oracle 12c OLTP on a VNXe3200. A test enables you to ensure this VSPEX solution meets your business requirements. Refer to the Design Guide to better understand test methodology before you verify your solution.
Two storage pools without FAST Cache enabled (one for Oracle datafiles and the other for redo log files) were built on the SAS 10K RPM disk to verify the baseline performance.
Key metrics
After you identify whether or not to use FAST Cache and the business requirements are satisfied, you must choose which Oracle 12c testing metrics to capture and the thresholds for each metric when running the Oracle Database 12c verification tests.
Refer to the VSPEX Sizing Tool to identify the Oracle Database 12c key performance metrics that are shown in Table 22.
Table 22. Example of key metrics for an Oracle Database 12c deployment
Metric Threshold
Average CPU utilization (%) Less than 80%
Average disk latency Less than 20 ms
The Oracle Database 12c virtual machine profile in Table 23 was used to verify that the compute and storage infrastructure satisfies the performance requirements.
Overview
Verify baseline performance
Chapter 6: Solution Verification
44 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Table 23. Oracle Database 12c virtual machine profile
Profile Value
Total Oracle database capacity 1 TB
Number of database instances 1
Number of virtual machines 1
Oracle Server virtual machine configuration
32 virtual processors (vCPUs) and 98 GB memory, with 64 GB reserved for the Oracle SGA and 32 GB reserved for the PGA
Table 24 shows the Oracle Database 12c storage pool design for the above profile. The design follows these best practices:
Distribute the datafiles and redo log files to different storage pools.
Use separate storage pools for Oracle archived logs and backup files, if necessary.
Table 24. Oracle Database 12c storage pool design example
Pool name RAID type
Disk type Disk capacity
No. of disks
Oracle database datafile pool RAID 5 SAS disks 10,000 rpm
600 GB 30
Oracle database log file pool RAID 10 SAS disks 10,000 rpm
600 GB 8
Test results
You need to interpret your results based on the key metrics after obtaining the test results. All the key metrics were achieved in the example shown in Table 25, which means the current Oracle 12c environment can meet the business requirements.
Table 25. Key performance metrics for Oracle Database 12c without FAST Cache
Performance metric Baseline performance
Threshold
Host CPU 35% Less than 80%
Host IOPS Up to 4,573 N/A
Datafile R/W response time 20 ms Less than or equal to 20 ms
The FAST Cache performance acceleration verification uses the same data file and log file pool as the baseline performance verification, with FAST Cache enabled on the data file pool only. The FAST Cache configuration for the
Verify FAST Cache performance acceleration
Chapter 6: Solution Verification
45 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
verification test used two 200 GB FAST Cache SSDs with RAID 1/0 and 183 GB of total usable disk space.
When you use FAST Cache to accelerate the Oracle Database 12c performance, the preferred application workloads are small-block random I/O applications with high frequency access to the application data.
Avoid enabling FAST Cache when the primary workload is sequential. The primary I/O of the Oracle redo log files is sequential, so we excluded the log from the FAST Cache verification.
Note: For more FAST Cache best practices, refer to the companion Design Guide.
Key metrics
After enabling FAST Cache on the existing database, the following performance improvement is expected:
Reduction of the I/O access latency
IOPS and TPS increments
Test results
After you get the test results, you need to interpret the results based on the key metrics. In the example shown in Table 26, all the key metrics were achieved, which means the current Oracle Database 12c environment can meet the business requirements.
Table 26. Key metrics for Oracle Database 12c with FAST Cache
Performance FAST CACHE performance Thresholds
Host CPU 75% Less than 80%
Host IOPS Up to 7426 N/A
Data R/W response time 7 ms Less than 20 ms
Performance comparison before and after FAST Cache enabled
Table 27 and Figure 9 compare database performance before and after FAST Cache was enabled to verify the FAST Cache performance acceleration. Compared to the baseline performance results shown in Table 25, adding two 200 GB FAST Cache flash drives to the VNXe3200 array increased Oracle Database 12c IOPS from 4,573 to 7,426, and response time was reduced from 20 ms to 7 ms.
Chapter 6: Solution Verification
46 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Table 27. Comparison before and after FAST Cache enabled
Performance metric Baseline FAST Cache
IOPS 4,573 7,426
Response time (ms) 20 7
Figure 9 shows the comparison between the baseline performance and the FAST Cache accelerated performance.
Figure 9. Performance comparison before and after FAST Cache enabled
Chapter 6: Reference Documentation
47 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Chapter 6 Reference Documentation
This chapter presents the following topics:
Reference documentation ........................................................................... 48
Links .......................................................................................................... 49
Chapter 6: Reference Documentation
48 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Reference documentation
The following documents are useful and relevant references for this solution.
The following documents are available on EMC Online Support or EMC.com:
EMC VSPEX Infrastructure as a Service with VMware vCloud Suite Design and Implementation Guide
EMC VSPEX Private Cloud VMware vSphere 5.5 for up to 200 virtual machines Enabled by Microsoft Windows Server 2012 R2, EMC VNXe3200, and EMC Powered Backup Proven Infrastructure Guide
EMC VNXe Series Configuration Worksheet
EMC VNXe3200 Installation Guide
Using EMC VNXe3200 System with VMware NFS or VMware VMFS
The following documentation is available on the Oracle documentation website:
Oracle Database Documentation Library 12c Release 1 (12.1)
Oracle Database Release Notes for Linux
The following articles or notes are available on My Oracle Support:
HugePages on Oracle Linux 64-bit (Article ID 361468.1)
FAQ: How to Use AWR Reports to Diagnose Database Performance Issues (Article ID 1359094.1)
The following documentation is available on the VMware website:
VMware vSphere vCenter Server Installation and Setup documentation
Performance Best Practices for VMware vSphere 5.5
VMware vSphere Networking documentation
VMware vSphere Storage documentation
VMware vSphere Resource Management documentation
Automating High Availability (HA) Services with VMware HA Technical Note
The following article is available in the VMware Knowledge Base:
Installing VMware Tools in a Linux virtual machine using Red Hat Package Manager (RPM) (Article ID 1018392)
EMC
Oracle
VMware
Chapter 6: Reference Documentation
49 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Links
The following web pages are additional references for this solution:
Everything Oracle at EMC: https://community.emc.com/community/connect/everything_oracle
My Oracle Support website: https://support.oracle.com (user account required)
General VMware documentation: http://www.vmware.com
vSphere documentation: http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html
vSphere PowerCLI documentation: http://www.vmware.com/support/developer/PowerCLI/
Note: The links provided in this guide were working correctly at the time of publication.
Chapter 6: Reference Documentation
50 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Appendix A: Configuration Worksheet
51 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Appendix A Configuration Worksheet
This appendix presents the following topic:
Configuration worksheet for VSPEX ............................................................. 52
Appendix A: Configuration Worksheet
52 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Configuration worksheet for VSPEX
Before configuring Oracle Database 12c for this solution, gather customer-specific configuration information, such as IP addresses, hostnames, and so on. The following tables provide a worksheet that you can use to record the information. You can also print and use the worksheet as a customer “leave behind” document for future reference.
To confirm the customer information, you can cross reference this worksheet with the relevant array configuration worksheet, such as: EMC VNXe Series Configuration Worksheet or VNXe Installation Assistance for File/Unified Worksheet.
Table 28. Common server information
Server name Purpose Primary IP address
Domain Controller
DNS Primary
DNS Secondary
DHCP
NTP
SMTP
SNMP
vCenter Server
Database server 01
Database server 02
Database server 03
Table 29. ESXi server information
Server name Purpose Primary IP address
Private net (storage) addresses
ESXi
Host 1
ESXi
Host 2
…
Appendix A: Configuration Worksheet
53 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection
Implementation Guide
Table 30. Array information
Array name
Array Name
Management IP
NAS admin password
SPA IP address
SPB IP address
Virtual machine infrastructure pool name
Virtual machine infrastructure file system name
Datafile storage pool name
Datafile file system name
Redo log storage pool name
Redo log file system name
FRA storage pool name
FRA file system name
Table 31. Network infrastructure information
Name Purpose IP address Subnet mask
Default gateway
Ethernet Switch 1
Ethernet Switch 2
…
Table 32. VLAN information
Name Network purpose VLAN ID Allowed subnets
Virtual Machine Networking Management
NFS networking
vMotion
Appendix A: Configuration Worksheet
54 EMC VSPEX for Virtualized Oracle Database 12c OLTP Enabled by EMC VNXe and EMC Data Protection Implementation Guide
Table 33. Service accounts
Account Purpose Password (optional, secure appropriately)
vCenter Server administrator
Array administrator
Oracle Database administrator