HPE Reference Configuration for VMware vSphere Metro Storage Cluster with HPE 3PAR Peer Persistence on HPE BladeSystem

Reference Architecture

Reference Architecture

Contents Executive summary ................................................................................................................................................................................................................................................................................................................................ 3 Solution overview ..................................................................................................................................................................................................................................................................................................................................... 3

Solution diagram ................................................................................................................................................................................................................................................................................................................................ 5 Solution components ............................................................................................................................................................................................................................................................................................................................ 5

HPE BladeSystem c7000 Enclosure................................................................................................................................................................................................................................................................................ 5 HPE Onboard Administrator ................................................................................................................................................................................................................................................................................................... 6 HPE Virtual Connect ...................................................................................................................................................................................................................................................................................................................... 7 HPE ProLiant BL460c Gen9 Server Blade ................................................................................................................................................................................................................................................................ 8 HPE 3PAR StoreServ Storage ............................................................................................................................................................................................................................................................................................... 8

System setup............................................................................................................................................................................................................................................................................................................................................ 10 HPE 3PAR Peer Persistence ............................................................................................................................................................................................................................................................................................... 12 Management port on HPE 3PAR StoreServ ......................................................................................................................................................................................................................................................... 14 VMware vSphere Metro Storage Cluster (vMSC) configuration ........................................................................................................................................................................................................ 14 Peer Persistence configuration ......................................................................................................................................................................................................................................................................................... 16 Deploying Quorum Witness ................................................................................................................................................................................................................................................................................................. 16 Remote Copy configuration ................................................................................................................................................................................................................................................................................................. 20 Changing Group Policy ............................................................................................................................................................................................................................................................................................................ 23 Exporting LUNs to hosts ........................................................................................................................................................................................................................................................................................................ 24

Failover/Switchover ............................................................................................................................................................................................................................................................................................................................ 25 Planned switchover ..................................................................................................................................................................................................................................................................................................................... 25 Recovering from automatic failover ............................................................................................................................................................................................................................................................................. 27 HPE 3PAR Peer Persistence manual transparent switchover ............................................................................................................................................................................................................. 27 Testing manual failover/switchover ............................................................................................................................................................................................................................................................................. 28

Summary ...................................................................................................................................................................................................................................................................................................................................................... 36 Appendix A: Bill of materials ...................................................................................................................................................................................................................................................................................................... 36 Resources and additional links ................................................................................................................................................................................................................................................................................................ 37

Reference Architecture Page 3

Executive summary Effective business continuity and disaster recovery can be the difference between the companies that survive or not, when disaster strikes. Very careful planning and technology deployment to support the plan are crucial in getting the business back to normal operations. Recovery objectives once measured in days are now measured in seconds or minutes. Information Technology is now the enabler of the business and with the increased requirements to enable the business, there is no tolerance for any downtime, whether it is a small or medium business or a large enterprise.

Disasters can occur anytime anywhere in your business environment. Natural disasters, such as earthquakes, tsunamis, hurricanes, fires, and floods can be devastating. Human error and hardware and software failure are just as unpredictable and expensive. Proactive disaster recovery planning and implementation is the best way to prevent loss. In the U.S. alone the number of disaster declarations by Federal Emergency Management Agency (FEMA) are increasing year by year. When a disaster strikes, your data must be accessible from an off-site location, for an unknown length of time. Staff may be limited and time will be important, so you need to know you are protected; every minute your system is down, the financial implications grow.

Addressing this crucial need, HPE 3PAR Peer Persistence software enables HPE Converged Infrastructure systems located in different data centers at metropolitan distances to act as peers to each other, presenting continuous storage system access to hosts and servers connected to them. This capability allows you to configure a high-availability solution between two sites or data centers where storage access failover and failback remain completely transparent to the hosts and applications running on those hosts. Compared to traditional cluster failover models where, upon failover, the applications must be restarted, Peer Persistence software allows hosts to remain online serving their business applications even when they switch storage access from their original site to the storage array on the disaster-recovery (DR) site, resulting in a much improved recovery time.

This paper provides information about deploying a VMware® vSphere® Metro Storage Cluster (vMSC) across two data centers or sites using HPE 3PAR Peer Persistence on HPE Converged Infrastructure. A vMSC configuration is designed to maintain data availability beyond a single physical or logical site.

Target audience: This document is intended for those seeking guidance in the implementation of a hardware based synchronous replication solution utilizing virtual machines and two separate storage arrays. This document assumes the reader has a basic understanding of several key data center technologies including, but not limited to: servers, storage, networking, power, solution management, virtualization, and hypervisors.

This white paper describes a project developed by Hewlett Packard Enterprise in February 2017.

Disclaimer: Products sold prior to the separation of Hewlett-Packard Company into Hewlett Packard Enterprise Company and HP Inc. on November 1, 2015 may have a product name and model number that differ from current models.

Solution overview HPE 3PAR Peer Persistence provides high availability Peer Persistence is a high availability configuration between two data centers in which the ESXi hosts are set up in a metro cluster configuration with access to storage arrays in both sites. Storage volumes created on one storage array are replicated to the other array using synchronous HPE 3PAR Remote Copy to ensure that the volumes are in sync at all times. Each volume will have a primary source copy and a backup target copy.

Peer Persistence utilizes Asymmetric Logical Unit Access (ALUA) capability that allows paths to a SCSI device to be marked as having different characteristics. With ALUA, the same LUN can be exported from both arrays simultaneously, but only the paths to the source (primary) volume will be marked as active. The paths to the target volume will be marked as standby preventing the host from performing any I/O using those paths.

Reference Architecture Page 4

In the event of a non-disruptive array volume migration scenario, the standby paths are marked as active, and host traffic to the primary storage array is redirected to the secondary storage array with minimal to no impact to the hosts. Figure 1 below shows a recommended Peer Persistence configuration.

Figure 1. Logical view of a Uniform vMSC configuration implemented with Peer Persistence

The Peer Persistence software supports bi-directional replication, which enables each site to perform as a disaster recovery center for the other. It enables non-disruptive automatic failover or manual switchover of applications from one site to another.

An example would be the use of VMware vMotion within a VMware vSphere environment. Peer Persistence provides the same stretched cluster configuration across data centers that are used with VMware’s vSphere Metro Storage Cluster certification to enable multi-site storage high availability. In Figure 1, a few virtual machines (VMs) are being serviced by an HPE 3PAR StoreServ storage system on site 1 while other VMs are being serviced by another HPE 3PAR StoreServ storage system at site 2 located within metropolitan distances from one another; vMotion allows customers to move VMs across sites. Peer Persistence software enables the movement of VMs across data centers to be completely transparent to the applications those VMs are running.

Peer Persistence achieves automatic transparent failover in the event of a complete array or data center failure with the help of the HPE 3PAR Peer Persistence Quorum Witness software. The HPE 3PAR Peer Persistence Quorum Witness software needs to be installed at a third site and will act as an independent witness should one of the arrays or data centers become unavailable.

Refer to http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA5-6753ENW for HPE 3PAR Peer Persistence in a vSphere environment.

Reference Architecture Page 5

Solution diagram The tested solution hardware for a single site is shown in the figure below. The HPE BladeSystem multi-rack solution provides high availability with redundant components. The solution can be grown within the racks or by adding additional compute, network, and storage components.

Figure 2. Overview of lab hardware setup for vMSC deployed on two HPE BladeSystem technology overview

Solution components HPE BladeSystem is a modular infrastructure platform that converges compute, storage, fabric, management and virtualization to accelerate operations and speeds delivery of applications and services running in physical, virtual, and cloud-computing environments. The unique design of the HPE BladeSystem c-Class helps reduce cost and complexity while delivering better, more effective IT services to end users and customers.

HPE BladeSystem c7000 Enclosure The HPE BladeSystem c7000 Enclosure represents an evolution of the entire rack-mounted infrastructure, consolidating and repackaging featured infrastructure elements – computing, storage, networking, and power – into a single infrastructure-in-a-box that accelerates data center integration and optimization.

Figure 3. HPE BladeSystem c7000 enclosure

Reference Architecture Page 6

The HPE BladeSystem enclosure infrastructure is adaptive and scalable. It transitions with your IT environment and includes modular server, interconnect, and storage components. The enclosure is 10U high and holds full-height and/or half-height server blades that may be mixed with storage blades, plus redundant network and storage interconnect modules. The enclosure includes a shared high-speed nonstop passive mid-plane with aggregate bandwidth for wire-once connectivity of server blades to network and shared storage. Power is delivered through a passive pooled-power backplane that enables the full capacity of the power supplies to be available to the server blades for improved flexibility and redundancy. Power input is provided with a very wide selection of AC and DC power subsystems for flexibility in connecting to data center power.

You can populate an HPE BladeSystem c7000 Enclosure with these components:

• Server, storage, or other optional blades

• Interconnect modules (four redundant fabrics) featuring a variety of industry standards including:

– Ethernet

– Fibre Channel

– Fibre Channel over Ethernet (FCoE)

– InfiniBand

– iSCSI

– Serial Attached SCSI (SAS)

• Hot-plug power supplies supporting N+1 and N+N redundancy

• HPE Onboard Administrator (OA) management module

HPE Onboard Administrator The HPE Onboard Administrator centralizes c-Class infrastructure management. Together with the enclosure’s HPE Insight Display, the Onboard Administrator has been designed for both local and remote administration of HPE BladeSystem c-Class components and provides the following capabilities:

• Wizards for simple, fast setup and configuration

• Highly available and secure access to the HPE BladeSystem infrastructure

• Security roles for server, network, and storage administrators

• Automated power and cooling of the HPE BladeSystem infrastructure

• Agent-less device health and status

• Thermal Logic power and cooling information and control

Figure 4. HPE Onboard Administrator for the BladeSystem c7000 chassis

Reference Architecture Page 7

HPE Virtual Connect HPE Virtual Connect technology provides wire-once, change-ready connectivity that is simple, flexible, and secure. This technology is a key element of HPE Converged Infrastructure, providing a better way to connect your virtualized environment to the network core. Rather than tying profiles to specific server blades, you create a profile for each of the bays in the c7000 enclosure; Virtual Connect then maps physical LAN or SAN connections to these profiles, allowing you to manage connectivity without involving LAN or SAN administrators. In addition, if a server blade were to fail, you could move its associated profile to a bay containing a spare blade, thus restoring availability without needing to wait for assistance.

HPE Virtual Connect FlexFabric technology supports the convergence of traffic based on different network protocols, allowing you to split a 20 Gb network connection into four variable partitions. The benefits of FlexFabric technology include the ability to replace multiple low-bandwidth physical NIC ports with a single port, lower management burden, fewer NICs and interconnect modules, and lower power and operational costs.

Key benefits of Virtual Connect:

• Servers are change-ready. Move, add or change servers without affecting the LAN and SAN

• Reduce cables without adding switches to manage

• Standards-based compatibility with other brands of data center networking infrastructure

• Pre-configure network connections for blade enclosure bays before servers are installed for easy, drop-in installation

Figure 5. HPE Virtual Connect FlexFabric-20/40 F8

HPE Virtual Connect FlexFabric 20/40 F8 Modules are the simplest, most flexible way to connect virtualized server blades to data or storage networks. HPE Virtual Connect FlexFabric 20/40 F8 Modules eliminate up to 95 percent of network sprawl at the server edge using one device that converges traffic inside enclosures and directly connects to external LANs and SANs. Utilizing Flex-20 technology with Fibre Channel over Ethernet (FCoE) and accelerated iSCSI, these modules converge traffic over the industry’s first high-speed 20 Gb connections to servers with HPE FlexFabric Adapters (HPE FlexFabric 20 Gb 2-port 630FLB and 630M Adapters). Each redundant pair of Virtual Connect FlexFabric modules provides eight adjustable downlink connections (six Ethernet and two Fibre Channel, or six Ethernet and two iSCSI, or eight Ethernet) to dual-port 20 Gb FlexFabric Adapters on servers. Up to eight uplinks are available for connection to upstream Ethernet (up to 40GbE) and Fibre Channel switches. Virtual Connect FlexFabric modules avoid the confusion of traditional and other converged network solutions by eliminating the need for multiple Ethernet and Fibre Channel switches, extension modules, cables, and software licenses. In addition, Virtual Connect wire-once connection management is built-in enabling server adds, moves, and replacements in minutes instead of days or weeks.

For more information on HPE Virtual Connect, visit: hpe.com/us/en/integrated-systems/virtual-connect.html

Reference Architecture Page 8

HPE ProLiant BL460c Gen9 Server Blade Designed for a wide range of configuration and deployment options, the HPE ProLiant BL460c Gen9 Server Blade provides the flexibility to optimize your core IT applications with right-sized storage for the right workload for a lower TCO.

HPE ProLiant BL460c Gen9 Server Blades are used in this solution for hosting various core SfB server roles. Designed for a wide range of configuration and deployment options, the HPE ProLiant BL460c Gen9 Server Blade provides the flexibility to optimize your core IT applications with right-sized external storage for the right workload to achieve a lower TCO. All of this is managed by HPE OneView, the converged management platform that accelerates IT service delivery and boosts business performance. The server, combined with Intel® Xeon® E5-2600 v3 processors and HPE DDR4 SmartMemory, provides improved performance as compared to previous generations.

Figure 6. HPE ProLiant BL460c Gen9 Server Blade

For more information on HPE ProLiant BL460c Gen9, visit: hpe.com/us/en/product-catalog/servers/proliant-servers/pip.hpe-proliant-bl460c-gen9-server-blade.7271227.html

HPE 3PAR StoreServ Storage HPE 3PAR StoreServ Storage offers the performance and flexibility that you need to accelerate new application deployments and support server virtualization, cloud, IT as a Service (ITaaS), and your future technology initiatives. It is a storage platform that allows you to spend less time on management, gives you technically advanced features for less money, and eliminates tradeoffs that require you to sacrifice critical capabilities such as performance and scalability. With HPE 3PAR StoreServ Storage, you can serve unpredictable and mixed workloads, support unstructured and structured data growth, and meet both file and block storage needs.

Simple to install, own, and upgrade • Software suites simplify purchasing and lower costs.

• HPE SmartStart and HPE 3PAR Rapid Provisioning get you up and running in minutes.

• Reconfigurable in just seconds without disruption.

• Shared management console across all HPE 3PAR StoreServ arrays simplifies administration.

• Manage Block, File, and Object access from a single interface for maximum agility.

Reference Architecture Page 9

• Self-diagnostic and “phone home” capabilities automate support.

• HPE 3PAR Online Import supports seamless migration from HPE EVA Storage and EMC Storage arrays

• Mesh-Active architecture: Fine-grained virtualization and system-wide striping.

• Purpose-built HPE 3PAR StoreServ ASIC: Supports mixed workloads with extremely high performance levels.

• HPE 3PAR Adaptive Read and Write: Matches host I/O size reads and writes to flash media at a granular level to avoid unnecessary data reads and writes to reduce latency.

• Autonomic cache offload: Reduces cache bottlenecks by automatically changing the frequency at which data is offloaded from cache to flash media based on utilization rate.

• Multi-tenant I/O processing: Enables performance improvement for mixed workloads by breaking large I/O into smaller chunks.

• HPE 3PAR Persistent Cache: Preserves service levels, so they are not impacted by unplanned component failures

• HPE 3PAR Persistent Ports: Allows non-disruptive upgrades without relying on multi-pathing software and without initiating failover.

• HPE 3PAR Peer Persistence: Ability to federate storage across data centers without being constrained by physical boundaries.

• HPE 3PAR StoreServ Data at Rest Encryption: Protects data from both internal and external security breaches.

• Flash-based media failure reconstruction: This enables the system to provide consistent performance levels even under situations of flash media failure.

• HPE 3PAR Remote Copy software enables low recovery time objectives (RTOs) and zero data-loss recovery point objectives (RPOs) with complete distance flexibility.

Thin-deduplication increases capacity efficiency without compromising performance • Inline deduplication lowers flash cost by saving capacity up front without the need for any storage pools or post-processing.

• Avoids unnecessary writes to flash media extending its life.

• Integration with Peer Motion increases storage efficiency across systems.

HPE 3PAR StoreServ storage is also backed by the Get 6-Nines Guarantee, which stands behind the ability of all quad-node and larger HPE 3PAR StoreServ Storage systems to deliver 99.9999 percent data availability.

For more information on HPE 3PAR StoreServ, visit: hpe.com/info/3par

Reference Architecture Page 10

System setup Initial setup steps Customers can deploy a vMSC configuration in separate data centers that are geographically dispersed or side by side. The following setup steps are performed after delivery of the two HPE BladeSystem environments.

• Bridge the appropriate networks between the two HPE BladeSystem environments.

• Merge the two SAN fabrics from each HPE BladeSystem.

• Verify that ESXi hosts from both HPE BladeSystem environments can be managed by a single VMware vSphere vCenter data center.

In Figure 7, the ESXi hosts along with their guest virtual machines utilize a network that is reachable from both sites. The HPE 3PAR StoreServ systems must also be reachable on the shared network.

Figure 7. VMware vSphere Web Client: Manage/Networking tab

Reference Architecture Page 11

Launching the HPE Storage Administrator Portal provides an input page to add storage arrays.

Figure 8. VMware vSphere Web Client: HPE Storage Systems

In Figure 9 the two HPE 3PAR StoreServ systems are configured to communicate on the same shared network used for the vMSC.

Figure 9. HPE Storage Administrator Portal for VMware vCenter

Reference Architecture Page 12

In Figure 10, the HPE Management Summary page shows that both HPE 3PAR StoreServ systems are being managed and provide important capacity information for Host and VMs. Implementation of vMSC on HPE 3PAR StoreServ with Peer Persistence requires that both arrays are accessible on the same network. ESXi hosts, VMs, and management devices need to be reachable across both sites on the same network, as well.

Figure 10. VMware vSphere Web Client: HPE Management/Storage/Summary

HPE 3PAR Peer Persistence Additional setup of the HPE 3PAR Peer Persistence solution includes the following requirements:

• Two HPE BladeSystem c7000 enclosures are needed.

• Firmware versions on both HPE 3PAR StoreServ 7400 arrays must be 3.1.2 MU2 or newer.

• The sites must be set up in an HPE 3PAR Remote Copy 1-to-1 configuration in synchronous mode.

• The round trip latency between the two sites must be 2.6 milliseconds or less.

• Quorum Witness software must be installed on a virtual machine at a third site.

• The HPE 3PAR StoreServ arrays communicate with the Quorum Witness via the management port over TCP/IP. The Quorum witness needs to be reachable from both sites.

• This non-disruptive failover configuration is supported with VMware ESXi 5.0 or newer hosts.

• All source and target volumes exported from the two HPE 3PAR StoreServ arrays may have the same volume WWN, but it is not required.

• Hosts have to be connected to the HPE StoreServ storage system using FC only (iSCSI and FCoE is not supported).

• The ESXi hosts that these volumes are exported to must be configured with host persona 11 (VMware), which supports host capability Asymmetric Logical Unit Access (ALUA) configuration considerations.

Reference Architecture Page 13

Refer to http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA5-6753ENW for the latest supported configuration and requirement for HPE 3PAR Peer Persistence in a vSphere environment

Host persona 11 Host personas are a set of behaviors that permit hosts connected to FC or iSCSI ports on the system to deviate from the default host behavior. By assigning a persona to a host, multiple host types that require distinct customized responses can share a single system port. For example, hosts running Windows®, Linux®, and IBM AIX operating systems can all connect to the same system port. This simplifies connecting hosts to the system and reduces management costs related to complex host connections.

Personal 11 supports ALUA path management and it must be used for host configurations that have ESXi hosts connected to both the primary array and the secondary array. Existing ESXi hosts defined with host persona 6 should be changed to persona 11 if these hosts are connected to both the primary array and the secondary array.

For host configurations that have independent ESXi hosts attached to the primary or secondary arrays, host persona 6 (Generic-Legacy) should be configured. This persona does not support Asymmetric Logical Unit Access (ALUA) path management.

Verify each VV has same WWN on both arrays It is important that the source and the target volumes on the HPE 3PAR StoreServ arrays have the same identity when both volumes are exported to the same ESXi host. Hence both volumes should have the same WWN. You can create volumes with the same WWN automatically using the admitrcopyvv command. See the HPE 3PAR Remote Copy Software User’s Guide for instructions on how to use the admitrcopyvv command.

The replicated volume WWN can also be changed later if the volumes were created using the HPE 3PAR StoreServ Management Console (CLI) or any method other than the admitrcopyvv.

Note Review the HPE 3PAR StoreServ Command Line Interface Administrator’s Manual or the HPE 3PAR StoreServ Management Console Online Help for instructions on how to edit a virtual volume’s (VV’s) WWN.

Verify VLUN status on both arrays Many customers still prefer to use the same LUN ID for management, documentation, and process reasons. Once the volumes are exported from both arrays, the showvlun command will show the state as “active” for VLUNs exported from the source side and “standby” for the VLUNs exported from the target side. ESXi hosts will send I/O requests only to the LUNs reported as active.

Quorum Witness The HPE 3PAR Peer Persistence Quorum Witness (QW) software enables transparent automatic failover between the two sites in a vMSC cluster. The HPE 3PAR Peer Persistence Quorum Witness software gets regularly updated with status information from the HPE 3PAR StoreServ arrays. In the event of a failure of one of the HPE 3PAR StoreServ arrays, the surviving HPE 3PAR StoreServ array detects that the Quorum Witness is not getting updated by the failed array and initiates a failover operation on any secondary groups associated with that failed array.

Note QW details have to be set up at the target level, and transparent failover will occur only for Remote Copy groups that use the target, are started and have the “auto_failover” policy set. Failover is not automatic for Remote Copy groups between the same two HPE 3PAR StoreServ systems which do not have the “auto_failover” policy set.

Reference Architecture Page 14

The Quorum Witness is typically set up on a third site where events that may impact site 1 or site 2 cannot impact the Quorum Witness site at the same time. Additionally, the QW connects to arrays on both sites using non Remote Copy links. With the above two configuration characteristics (site and link independence), QW helps determine the following nature of failures. Review the table below on expected behavior.

Table 1. Quorum Witness expected behavior

Failure Type Expected Behavior

Link Failure The QW can detect if the two arrays are alive but not communicating because of a link failure. QW would still be receiving updates from both of the arrays.

Array/Site failure The QW can detect when one of the arrays/sites fails. QW would not be receiving updates from one of the arrays that has failed.

The Quorum Witness software is packaged as a VMware Open Virtualization Format (OVF) and needs to be imported on an ESXi host. Only ESXi 5 and higher are supported. The Quorum Witness virtual machine should not be stored on a datastore allocated from either of the arrays that are part of the Peer Persistence configuration. If the datastore is located on one of the arrays that is part of the Peer Persistence configuration, failure of the array will make the datastore unavailable which will result in the shutdown of the Quorum Witness virtual machine. The Quorum Witness software uses port 8080 to communicate with the two HPE 3PAR StoreServ systems in the quorum. Hence firewall rules need to be changed to open port 8080 at the QW site.

The Quorum Witness is a passive component of the configuration, and the software itself does not provide any high availability, however, the virtual machine can be made highly available by using VMware HA.

Management port on HPE 3PAR StoreServ The HPE 3PAR StoreServ systems communicate with the Quorum Witness using the HPE 3PAR StoreServ management interface. The network should be set up such that the QW server has network connectivity to the admin interface on both arrays. It is required to connect the administration port from a minimum of two controller nodes to the network. However, if the array has more than two controllers the best practice is to connect all controller nodes to the network.

VMware vSphere Metro Storage Cluster (vMSC) configuration VMware vSphere Metro Storage Cluster is a certified configuration for stretched storage cluster architectures. A vMSC configuration is designed to maintain data availability beyond a single physical or logical site. A storage device configured in the vMSC configuration is supported after successful vMSC certification. All supported storage devices are listed in the VMware Storage Compatibility Guide.

A tested and certified VMware vSphere Metro Storage Cluster using HPE 3PAR StoreServ Storage is supported in accordance with the VMware description of a “uniform access” vMSC configuration. Particular to the uniform access configuration, host data path connections at each site are cross-connected to the peer data center site storage array. ESXi hosts access volumes in the data center local array via active paths. Connectivity to standby peer volumes on the distance array is maintained in standby mode until such time as a failover or switchover. In concert with HPE 3PAR Peer Persistence, Quorum Witness and with Automated Transparent Failover (ATF) enabled, a minimally disruptive switchover or failover of volume access across sites can be achieved.

Two vMSC configurations are possible based on how the hosts are connected to the storage arrays. Uniform is the recommended configuration for Automated Transparent Failover. The tested configuration for this paper utilized a vMSC Uniform configuration with two HPE BladeSystem environments.

Reference Architecture Page 15

Uniform In this configuration each host can access LUNs exported from both the sites. Figure 1 shows a typical Uniform configuration. This setup helps in protecting against a storage failure at a site. So, if the array on site 1 fails, storage will failover to the HPE 3PAR StoreServ array on site 2 transparently and automatically allowing VMs on site 1 continued access to storage on site 2 without any interruption. Of course this implies that customers need to have good bandwidth between their sites. Outside of what happens in sync replication (ask only after write to remote array is completed), even host to storage array traffic will be routed across the FC connections to the remote site (in synchronous replication, hosts are writing only to the local array).

Uniform host access provides:

• Protection against storage failure

• Active load balancing

Note When deployed in a vMSC Uniform configuration, active load balancing of the secondary paths/volumes are made active on site 2 array as a means to achieve load balancing. Load balancing provides additional benefits in addition to the Automated Transparent Failover that a Uniform configuration provides.

Table 2. Testing scenarios and expected results

Scenario HPE 3PAR StoreServ Storage System Behavior VMware HA Behavior

Single Array Host Path Failure Hosts use alternate paths to the surviving array nodes at the site and maintain volume access

No effect observed

Single Array Node Failure No effect. Inter-site connectivity is maintained by alternate link

No effect observed

Single Storage Inter-Site Link Failure Peer volume synchronization is disabled and Automated Transparent Failover is disabled

No effect observed

Quorum Witness Failure Automated Transparent Failover is disabled No effect observed

Simultaneous Quorum Witness and Storage Inter-Site Links Fail

Peer volume synchronization is disabled and Automated Transparent Failover is disabled

No effect observed

Single Site Storage Array Failure Automated Failover occurs and Peer volumes and paths are made active on the surviving site

No effect observed

Complete Site Failure Automated Failover occurs and Peer volumes and paths are made active on the surviving site

Virtual machines are restarted on ESXi hosts on the surviving site

Note For additional details review the VMware Knowledge Base article: Implementing vSphere Metro Cluster using HPE 3PAR StoreServ Peer Persistence.

Non-Uniform In this configuration each host can access storage resources available only on its local site. This is geared towards protecting against a complete site failure similar to what VMware vCenter Site Recovery Manager (SRM) provides.

The concept here is that if a complete site fails (storage and server cluster), Peer Persistence will be able to transparently and automatically failover the storage to site 2 while the host application fails over in parallel. Consider this really as a DR solution. Non-uniform configurations will not allow an application to transparently failover. If a switchover is executed, the local ESXi servers will lose connectivity, the VMs will go offline and will be switched according to the HA setup, if they are switched to the remote site, they will restart automatically.

Reference Architecture Page 16

Note With HPE 3PAR OS 3.1.2 MU2 and later, HPE supports vMSC uniform configurations only. Non-Uniform configurations are not supported.

Additional content relevant to planning and deploying a vSphere Metro Storage Cluster with HPE Peer Persistence is included in the following sections.

Peer Persistence configuration This section provides the steps to set up a single Remote Copy group in Peer Persistence configuration. Many of the details in the sections below were tested during the installation of vSphere Metro Storage Cluster using HPE 3PAR Peer Persistence on HPE BladeSystem.

Note Refer to the HPE 3PAR Remote Copy Software User Guide for more details and the complete set of options available for the commands used in this section.

Deploying Quorum Witness The HPE 3PAR Peer Persistence Quorum Witness software provides the arbitrator functionality in the Peer Persistence setup. For correct operation, and to avoid unnecessary automatic failovers, the Quorum Witness should be located on a third site, where events that might impact site 1 or site 2 cannot impact the Quorum Witness site at the same time. In addition, the Quorum Witness is connected to arrays on both sites using non-Remote Copy links, so that issues with Remote Copy links cannot impact connectivity with the Quorum Witness.

1. Start the vSphere client. In the IP address / Name drop-down menu, select the IP address or name of the vSphere host. Make sure you know the user name and password for the ESXi host to connect with your vSphere client.

2. In vSphere, select Deploy OVF Template from the File drop-down menu.

3. In the Deploy OVF Template pane, in the Deploy from a file or URL drop-down menu, select the .ovf file as the source location.

4. In the Open pane, make sure the .ovf file is selected, and click Open. The OVF Template Details screen displays.

5. In the Name field of the Deploy OVF Template pane, type a name for the Quorum Witness. The name you assign it does not affect Quorum Witness operation.

6. At least 250 GB disk space is recommended for Quorum Witness VM. In the Disk Format pane of the Deploy OVF Template pane, select the Thin Provision option.

7. When the VM is ready to complete, a pane showing deployment settings displays, indicating:

OVF file Download size Size on disk Name Host/Cluster Datastore Disk provisioning Network mapping

8. Click Finish. The Quorum Witness opens on the Getting Started tab.

Reference Architecture Page 17

Note Document the IP address displayed in the Summary tab after powering on the virtual machine. The IP address will be required input during the Quorum Witness configuration.

Figure 11. Find the IP address of the Quorum Witness server

Starting Quorum Witness Click on the Peer Persistence Configuration tab after selecting the Remote Copy Configuration navigation tree node. Use the IP address of the Quorum Witness that was noted down earlier. Do not click on the “Start Witness” checkbox at the time of creating the configuration. Press OK after specifying the IP address.

Figure 12. HPE 3PAR Management Console Remote Copy interface to start Quorum Witness

Reference Architecture Page 18

The Quorum Witness will have an initial state of Initializing. This state will automatically transition to Not-started when the stability of the system has been successfully confirmed. After the Quorum Status has changed to Not-started, then select both targets in the list and click on the Start button to start the Quorum Witness.

Figure 13. Quorum Witness status will change from Not-started to Started after completing initialization

The Quorum Witness status is available in the HPE 3PAR Management Console and has changed Quorum Status to Started.

Figure 14. HPE 3PAR Management Console Remote Copy: Remote Copy Configuration Quorum Witness Status

From the CLI issue the showrcopy command with the -qw option, and filter for target information. Q-Status is displayed. In Figure 15, the quorum status is started.

Figure 15. HPE 3PAR CLI can be used to check status of the Quorum Witness

Reference Architecture Page 19

Note Refer to the “HPE 3PAR Command Line Interface Administrator’s Manual” that matches the HPE 3PAR OS version for additional details and functionality. version 3.1.2, version 3.3.1

Troubleshooting Quorum Witness communication issues This operation needs to be done using the HPE 3PAR Command Line Interface. Log in to one of the HPE 3PAR StoreServ arrays and perform the following:

Find the target name using showrcopy command:

Figure 16. Showing remote copy information

Note Use the witness check command to verify the communication between Quorum Witness and the management interface of the two arrays. The two commands will not report any message on success.

Figure 17. Verify Quorum Witness communication between two arrays

The following is an example status return if the Quorum Witness server is unreachable:

Figure 18. Showing Quorum Witness communication failure between two arrays

Reference Architecture Page 20

Remote Copy configuration Remote Copy configurations and operations are based on a linked pair of storage systems (two HPE 3PAR StoreServ 7400), which are called a remote-copy pair.

In a remote-copy pair, one system is the primary system and one is the backup system. The primary system holds the primary virtual volumes, and the backup system holds the secondary virtual volumes. Secondary virtual volumes contain copies of the primary virtual volumes. The volume that is part of a Primary Remote Copy group is the source volume. The replicate volume in the secondary Remote Copy group is called the target volume.

When you set up HPE 3PAR Remote Copy, you configure the backup system in the remote-copy pair as the target of the primary system, and the primary system as the target of the backup system.

Note The steps below assume that hardware requirements for Remote Copy are met, and ports, cabling and zoning are correctly set up; and also, that all requirements for Peer Persistence have been met.

Figure 19. HPE 3PAR Management Console New Remote Copy Configuration

Reference Architecture Page 21

Select the ports that will be used for Remote Copy. RCFC ports were used in the configuration tested for the HPE BladeSystem.

Figure 20. HPE 3PAR Management Console Remote Copy Ports

Next, in the New Remote Copy Configuration window as shown in Figure 21 below:

Create links between systems 1. Click a port on one system icon and drag the dotted line to the desired port on the target system icon.

2. A successful link is represented by a solid line between the ports.

3. Link at least two ports on each system to two ports on each target system.

Verify the Port Parameters 1. Select each link in the configuration diagram to view the linked port's parameters.

2. Verify the IP settings for each port and update parameters as necessary.

Figure 21. HPE 3PAR Management Console New Remote Copy Configuration Create Links between systems

3. Click on Finish as we would like to create groups later.

Reference Architecture Page 22

Creating Remote Copy groups Start the Create Remote Copy group wizard and select the source and target system. Next, specify the group name and select synchronous mode. In the figure below, the Specify Group inputs include:

• Source System: sstor-McCoy

• Source Group: MetroGroup05

• Backup Target Name: sstor01-Scotty

• Mode: Synchronous

• Option to Start group after competition

Figure 22. HPE 3PAR Management Console Create Remote Copy Group: Specify Group.

On the next screen, select the source volume to replicate and then select the “Create new volume” on the target system. It is preferred to create new volumes on the target, this way the volumes will have the same characteristics as the source volumes. If you manually create a volume beforehand, be sure that the volume size on the target matches the volume size on the source and both volumes have the same WWN.

Figure 23. HPE 3PAR Management Console Create Remote Copy Group: Virtual Volumes.

Reference Architecture Page 23

Add all volumes in the group and click Finish. The group state status will change to “started”, and the initial synchronization will begin. Additional groups can be created to meet business needs. A typical uniform vMSC configuration will have Remote Copy groups replicating in both directions.

Changing Group Policy The replication group policy must be set to “auto failover” to enable automatic transparent failover. Click on the Enable/Disable Groups button in the Peer Persistence tab. Select the group that you want to enable automatic failover for and press OK to continue.

Figure 24. HPE 3PAR Management Console Remote Copy: Enable Automatic Failover

Reference Architecture Page 24

Exporting LUNs to hosts When creating ESXi hosts they need to be created using Persona 11, for best practice add all hosts in the cluster in a host set. And add all VVs in a Remote Copy group in a VV set.

Figure 25. HPE 3PAR Management Console ESXi host Summary

After exporting the VVs from both arrays, press “Rescan Storage” from the “All vCenter Actions” drop-down for all ESXi hosts in the cluster.

Verify in the path details of the LUNs that the expected number of Active and Standby paths are available.

Figure 26. VMware vSphere Web Client ESXi host Manage Storage Paths view

Reference Architecture Page 25

Now you are ready to create datastores on these LUNs and start using them in the vSphere cluster.

Failover/Switchover Planned switchover In the example below, the CLI is used to initiate a planned failover/switchover. The Remote Copy group primary storage is currently on the sstor01-McCoy system and replicating synchronously to the sstor01-Scotty system (target).

Figure 27. Showing Remote Copy group information

Reference Architecture Page 26

The switchover command is executed successfully below. The primary role is moved to sstor01-Scotty and the backup/target role to sstor01-McCoy.

Figure 28. Executing switch over command

Reference Architecture Page 27

Recovering from automatic failover If one array has unplanned downtime, then the primary Remote Copy groups from that array that have “auto failover” policy enabled will be automatically failed over to the second array. A recovery process is needed to restart replication and reverse replication back in the original direction. After the determination of the unplanned failover is complete and both systems are online, the Remote Copy groups can perform a switchover to resume normal operations.

Figure 29. HPE 3PAR Management Console Remote Copy Recover option

The Restore button will be enabled after the recovery has completed. However, performing the restore operation might lead to ESXi hosts losing access to the LUNs for a short duration. The recommendation is to use the CLI commands to complete the failback. Log in to the array with the Primary-rev role and issue the following commands:

• showrcopy

• setrcopygroup reverse

• setrcopygroup switchover

• showrcopy

HPE 3PAR Peer Persistence manual transparent switchover The HPE 3PAR Peer Persistence switchover operation migrates host I/O from one storage array to the other and reverses the direction of data replication. A manual switchover operation is designed to facilitate service optimization and storage system maintenance activities within a high-availability data storage solution.

In a manual transparent switchover the operation can be initiated several ways:

• HPE 3PAR Command Line Interface setrcopy switchover command to a remote-copy volume group on the primary storage system

• Management console switchover command

Reference Architecture Page 28

The outcome of both the manual and the automatic peer-persistence operations will be the same, in that the original secondary system will begin to service host I/O to the failed-over volume groups; but in an automatic failover, replication will have stopped and will remain unavailable until the failed system is recovered.

Testing manual failover/switchover HPE 3PAR Peer Persistence manual switchovers were tested on the HPE BladeSystem. HPE Management is integrated within the VMware vSphere Web Client and provides storage administrators an interface to manage the vSphere Metro Storage Cluster. The HPE 3PAR Management Console along with HPE 3PAR CLI provide storage administrators with intuitive interfaces to manage the switchovers. In the section below switchover testing is discussed. Several virtual machines (VMs) running Microsoft® SQLIO and Microsoft Jetstress were utilized during the switchover testing. These tests are discussed in detail in the following sections.

Figure 30. VMware vSphere Web Client HPE Management Storage Replications tab

Reference Architecture Page 29

The vSphere Web Client HPE Management plug-in enables access to HPE OneView and also integrates monitoring and managements tasks. Source disk name, source array and replica/target arrays provide replication direction at a glance. Selecting the source disk name opens an interface to invoke a manual switchover. After selecting switchover, additional status is available.

Figure 31. HPE Storage Management plug in for the vSphere Web Client enables Peer Persistence Switch replication switch

The HPE 3PAR Management Console provides Remote Copy switchover at the Group level. As shown in the figure below, the manual switchover migrates the Remote Copy group from the primary to the secondary array without impacting the host I/O. This command requires that the associated hosts are connected to both the primary and the secondary arrays.

Figure 32. HPE 3PAR Management Console Remote Copy Switchover

Reference Architecture Page 30

In the Management Console screen in Figure 32, the Remote Copy group MetroGroup01 is selected for a manual switchover. Selecting OK will bring up a confirmation screen for the Remote Copy Switchover System command.

Figure 33. Remote Copy Switchover System Window

Selecting “Yes” will immediately initiate the Remote Copy Switchover. Selecting “No” closes the confirmation window and provides the Cancel option. Selecting “Cancel” returns the focus to the MC.

Figure 34. Switchover Remote Copy Group executed in Remote Copy Configuration

Reference Architecture Page 31

MetroGroup01 containing the virtual volumes McCoy_OS01 and McCoy_DATA01 have changed their source and target storage arrays. The HPE 3PAR Peer Persistence switchover operation has migrated host I/O from sstor01-McCoy to sstor01-Scotty and reversed the direction of data replication. HPE 3PAR Remote Copy ensures that the data in the virtual volumes within a volume group maintain write consistency at all times. The switchover operation is transparent to the hosts.

SQLIO activity during manual switch of source/target replication traffic Microsoft provides an easy to use I/O simulator called SQLIO Disk Subsystem Benchmark Tool. The tool can simulate several types of workloads, including typical SQL Server workloads. Several of the vMSC VMs ran SQLIO loads for baseline and during manual switchover.

The test below was run for 900 seconds (15 minutes) and generated random 8K read IOPS:

C:\SQLIO\SQLIO.EXE -s900 -kR -frandom -b8 -t8 -o16 -LS -BN e:\TestFile.DAT

– sqlio v1.5.SG

– 8 threads reading for 900 seconds from file e:\TestFile.DAT

using 8KB random I/Os

enabling multiple I/Os per thread with 16 outstanding

buffering set to not use file nor disk caches (as is SQL Server)

using current size: 102400 MB for file: e:\TestFile.DAT

The 500 GB volume E: was over 90% full of data files.

Figure 35. SCOTTYAPP04 VM volume properties for SQLIO test

Reference Architecture Page 32

The 500 GB volume E$ was pre-written with data prior to running SQLIO. Each of the VMs has a DATA volume that is a Thin Provisioned Virtual Volume (TPVV) presented from the HPE 3PAR Storage System. As shown in the figure below the Management Console shows the TPVV has a capacity of 1024 GB and has written 36% of it. The SQLIO test was used to capture Host and storage IOPS and service time during a manual switchover.

Figure 36. HPE 3PAR Management Console: Provisioning/Virtual Volumes

The SQLIO 8K random read test generated high IOPS and low latency as shown below. The Avg. Disk Bytes/Read provides the average I/O size of 8K. The Disk Reads/sec and Disk Transfers/sec show the same 59,399 IOPS generated.

Figure 37. SQLIO Performance Monitor for SCOTTYAPP04 VM

Reference Architecture Page 33

The host read latency was 2 ms during the 4 replication switchover actions. In the figure below the HPE 3PAR Management Console Performance and Report Manager switchovers are clearly visible but do not impact host IOPS.

Figure 38. HPE 3PAR Management Console: Performance and Report IOs/Sec

The Performance and Reports Manager within the HPE 3PAR Management Console enables performance monitoring of system components such as disks, ports, CPUs, Virtual Volumes (VVs), and Virtual LUNs (VLUNs). Remote Copy Performance characteristics are included.

In summary, the SQLIO generated OLTP random IOPS that were serviced with no impact during 4 manual switchovers initiated from the HPE 3PAR Management Console.

Jetstress 2013 activity during manual switchovers Jetstress is a Microsoft tool primarily used to validate physical storage systems by simulating Exchange database I/O load. Jetstress has been optimized to evaluate the storage subsystem and not the host performance. Jetstress analyzes the performance data generated from a test and performs a reporting and verification process that compares the observed performance results against a set of acceptable values. During the test, binary performance data is written out to a .blg file. The primary goal of this test was to validate if Jetstress would pass the performance test after invoking a manual switchover of the storage source and target LUNs.

Jetstress evaluates latency values for Database Reads and Log writes.

Performance Test “Strict” mode (<= 6 hour test):

• Average Database Read Latency: 20ms

• Average Log File Write Latency: 10ms

• Max Database Read Latency: 100ms

• Max Log File Write Latency: 100ms

Jetstress will also detect I/O errors and report any errors in both the test log and error counts per volume table. The default Jetstress data sample time is 15 seconds. To capture any impacts from the test, sample time was modified in the Jetstress test XML file to a 1 second sample. The .blg file grows quickly at this granular collection interval but allows for manual analysis after the test to determine any latency spikes:

• Default Jetstress sample rate is 15 seconds: <LogInterval>15000</LogInterval>

• Switchover test modified to 1 second: <LogInterval>1000</LogInterval>

Reference Architecture Page 34

The Jetstress .blg file was loaded in Performance Monitor and provided the steady state transactions and two switchover operations. The logging interval of 1 second provided a granular capture of data and avoided masking impacts to the host during the test.

Figure 39. Jetstress IOPS displayed from Performance Monitor for SCOTTYAPP01 VM

The Performance and Reports Manager real time monitoring of both storage systems shows the switchover and switchback of source and target volumes. The host running Jetstress was not impacted by the switchovers and passed the storage verification test with no errors.

Figure 40. HPE 3PAR Management Console: Performance and Report IOs/Sec

Reference Architecture Page 35

The report view of Performance Monitor provides a real time snapshot of the Jetstress IOPS on the storage system. The SCOTTYAPP01 virtual machine has a 500 GB formatted volume from a 1 TB presented VLUN. There are 4 separate databases on the host volume along with their transaction logs. The 4 databases consume 412 GB of capacity.

Figure 41. Jetstress IOPS displayed from Performance Monitor report view for SCOTTYAPP01 VM

Jetstress analyzes the performance data generated from a test and performs a reporting and verification process that compares the observed performance results against a set of acceptable values. The Performance Test Result Report shows that the disk subsystem test passed with no impact from the two manual switchovers of replication traffic during the test. The Achieved Transactional I/O per Second includes only database reads and writes that are serviced from physical disk. Additional IOPS will be observed as total transactional I/O from the host performance monitor .blg file or from the actual storage system. Background database maintenance, transaction log I/O is not included in the Achieved Transactional I/O.

Figure 42. Jetstress 2013 Disk Subsystem Performance Test Result Report

Reference Architecture Page 36

Summary This paper provided implementation of a vSphere Metro Storage Cluster (vMSC) across two data centers or sites using HPE 3PAR Peer Persistence on HPE BladeSystem. VMware vSphere Metro Storage Cluster is a certified configuration for stretched storage cluster architectures. A vMSC configuration is designed to maintain data availability beyond a single physical or logical site.

HPE 3PAR Peer Persistence software enables HPE BladeSystem environments located in different data centers at metropolitan distances to act as peers to each other, presenting continuous storage system access to hosts and servers connected to them. This capability allows you to configure a high-availability solution between two sites or data centers where storage access failover and failback remains completely transparent to the hosts and applications running on those hosts. Compared to traditional cluster failover models where, upon failover, the applications must be restarted, Peer Persistence software allows hosts to remain online serving their business applications even when they switch storage access from their original site to the storage array on the disaster recovery (DR) site.

This paper provided tested results of the HPE 3PAR Peer Persistence switchover operation. The transparent switchover migrates host I/O from one storage array to the other and reverses the direction of data replication. A manual switchover operation is designed to facilitate service optimization and storage system maintenance activities within a high-availability data storage solution. Testing showed that the hosts maintained workloads of SQLIO and Jetstress without impact.

Appendix A: Bill of materials The HPE BladeSystem is a defined solution with the ability to vary key components to meet a customer’s needs. The product ordering information is defined in detail in the CS700 QuickSpecs in the “How to order” section of that document. This list notes the key customizations used for testing this solution.

(Per Site) HPE ConvergedSystem 700 (CS700) consisting of:

• HPE BladeSystem c7000 enclosure

– 2 x HPE Virtual Connect FlexFabric-20/40 F8 Module

– 2 x HPE Onboard Administrator

• 8 x HPE ProLiant BL460c Gen9 server blades

– 2 x Intel Xeon E5-2690 v3 12-core CPUs @ 2.60 GHz

– 256GB memory

– HPE Virtual Connect 10GbE FlexFabric

• 2 x HPE ProLiant DL360 Gen9 (management servers)

– 2 x Intel Xeon E5-2680 v3 12-core CPUs @ 2.50 GHz

– 256GB memory

• HPE 3PAR StoreServ 7400 4-node

– 120 x 300GB SFF Fast Class (FC) in 10 SFF Drive enclosures

• Switches

– 2 x HPE SN6000B Fibre Channel Switches

– 2 x HPE 5900AF-48XG-4QSFP+ Network Switches

– 2 x HPE 5900AF-48G-4XG-2QSFP+ Network Switches

Reference Architecture Page 37

Sign up for updates

© Copyright 2015-2018 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.

VMware and vSphere are registered trademarks of VMware, Inc. in the United States and/or other jurisdictions. Microsoft and Windows are registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Intel and Xeon are trademarks of Intel Corporation in the U.S. and other countries. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

4AA5-8914ENW, April 2018, Rev. 2

Resources and additional links HPE Reference Architectures, hpe.com/info/ra

Contact Hewlett Packard Enterprise, hpe.com/us/en/contact-hpe.html

HPE BladeSystem, hpe.com/us/en/integrated-systems/bladesystem.html

HPE Servers, hpe.com/servers

HPE ProLiant BL460c, hpe.com/servers/bl460c

HPE Storage, hpe.com/storage

HPE 3PAR Peer Persistence Software, hpe.com/us/en/product-catalog/storage/storage-software/pip.hpe-3par-peer-persistence-software.5335710.html

HPE Networking, hpe.com/networking

HPE Converged Infrastructure Management (OneView), hpe.com/info/hpeov

HPE Technology Consulting Services, hpe.com/us/en/services/consulting.html

VMware document, vmware.com/support/pubs

To help us improve our documents, please provide feedback at hpe.com/contact/feedback.