implementing microsoft multisite clustering using hp 3par peer persistence

Technical white paper

Implementing Microsoft multisite clustering using HP 3PAR Peer Persistence

Table of contents Introduction .................................................................................................................................................................................... 3

Terminology ................................................................................................................................................................................... 3

Features and benefits .................................................................................................................................................................. 3

Requirements ................................................................................................................................................................................. 4

Host persona 15 ........................................................................................................................................................................ 4

Verify each VV has same WWN on both arrays .................................................................................................................. 5

Verify VLUN status on both arrays ........................................................................................................................................ 5

Quorum Witness ........................................................................................................................................................................ 5

Planned switchover ...................................................................................................................................................................... 6

HP 3PAR Peer Persistence failures handling .......................................................................................................................... 6

Array-to-array communication failure ................................................................................................................................. 6

Single site to QW communication failure ............................................................................................................................. 6

Site 1 to QW and array-to-array communication failure.................................................................................................. 6

Site 2 to QW and array-to-array communication failure.................................................................................................. 6

Site 1 and site 2 both lose communications to QW ........................................................................................................... 7

Site 1 and site 2 to QW and array-to-array communication failure .............................................................................. 7

Multisite Windows Server Failover Cluster and HP 3PAR Peer Persistence considerations ......................................... 7

WSFC quorum model ............................................................................................................................................................... 7

Cluster validation test .............................................................................................................................................................. 8

Networking consideration ....................................................................................................................................................... 8

Tuning of cluster heartbeat settings .................................................................................................................................... 8

Microsoft MPIO DSM ................................................................................................................................................................. 8

HP 3PAR Quorum Witness hosting ....................................................................................................................................... 8

Remote Copy replication links ................................................................................................................................................ 8

HP 3PAR Peer Persistence maximums ................................................................................................................................ 9

Sample Peer Persistence configuration ................................................................................................................................... 9

Deploying Quorum Witness .................................................................................................................................................... 9

Requirements of Quorum Witness ........................................................................................................................................ 9

QW installation .......................................................................................................................................................................... 9

Creating Remote Copy targets ............................................................................................................................................. 10

Technical white paper | Implementing Microsoft multisite clustering using HP 3PAR Peer Persistence

Peer Persistence configuration ............................................................................................................................................ 12

Troubleshooting Witness Quorum Communication issues ........................................................................................... 13

Creating Remote Copy groups ............................................................................................................................................. 14

Changing Group Policy ........................................................................................................................................................... 15

Exporting LUNs to hosts ........................................................................................................................................................ 17

Planned switchover example ............................................................................................................................................... 22

Recovering from automatic failover ................................................................................................................................... 27

For more information ................................................................................................................................................................. 34


3

Introduction

HP 3PAR StoreServ Peer Persistence software enables HP 3PAR StoreServ systems located in different data centers at metropolitan distances to act as peers to each other, presenting nearly continuous storage system access to hosts connected to them. This capability allows users to configure a high-availability solution between two sites or data centers where storage access, failover, and failback remain completely transparent to the hosts and applications running on those hosts. Compared to traditional cluster failover models where, upon failover, the applications must be restarted, Peer Persistence software allows hosts to remain onlineserving their business applications even when they switch storage access from their original site to the storage array on the remote site, resulting in a much improved recovery time.

This paper provides information about deploying a Microsoft multisite cluster using Windows Server Failover Clustering (WSFC) across two data centers or sites with HP 3PAR StoreServ Peer Persistence. A Windows multisite cluster configuration is designed to maintain data availability beyond a single physical or logical site failure.

Terminology

Throughout this paper, we identify the volume that is part of a primary Remote Copy group as the source volume. The replicated volume in the secondary Remote Copy group is called the target volume. Volumes created on an HP 3PAR StoreServ are called virtual volumes, or VVs. A VV exported to the host is known as a virtual logical unit number (VLUN). A zoning operation creates a logical connection in the Fibre Channel (FC) SAN between an FC host bus adapter (HBA) on a server and one on a storage system.

Features and benefits

Peer Persistence is a high-availability configuration between two data centers in which Windows hosts are set up in a multisite cluster configuration with access to storage arrays in both sites. Storage volumes created on one storage array are replicated to the other array using synchronous Remote Copy to make sure that the volumes are in sync at all times. Peer Persistence software takes advantage of the Asymmetric Logical Unit Access (ALUA) capability that allows paths to an SCSI device to be marked as having different characteristics. With ALUA, the same LUN can be exported from both arrays simultaneously, but only the paths to the side accepting write to the volume will be marked as active. The paths to the secondary side volume will be marked as standby preventing the host from performing any I/O using those paths. In the event of a non-disruptive transparent failover scenario, the standby paths are marked as active, and host traffic to the primary storage array is redirected to the secondary storage array without impact to the hosts. Figure 1 shows a Peer Persistence configuration.

The Peer Persistence software supports bi-directional replication, which enables each site to perform as a disaster recovery center for the other. It enables users to move their applications from one site to another based on their business and performance needs without impact on the applications running on those hosts.

An example would be workload mobility using Microsoft Hyper-V Live Migration in a Hyper-V deployment under Windows Server Failover Cluster. Peer Persistence provides virtual machine (VM) mobility across data centers that are used with Windows Server Failover Clustering to enable multisite storage high availability. In figure 1, certain VMs are being serviced by an HP 3PAR StoreServ Storage system on site 1 while other VMs are being serviced by another HP 3PAR StoreServ Storage system at site 2 located within a metropolitan distance. Hyper-V Live Migration allows customers to move VMs across sites. Peer Persistence software presents the VMs with the same virtual volume after it moves across data centers. In other words, movement of VMs across data centers becomes completely transparent to the applications those VMs are running.

Peer Persistence achieves automatic transparent failover in the event of a complete array or data center failure with the help of the HP 3PAR Quorum Witness (QW) software. The HP 3PAR Quorum Witness software needs to be installed at a third site and will act as an independent witness should one of the arrays or data centers become unavailable.


4

Figure 1. Peer Persistence setup with a Microsoft multisite failover cluster configuration

Requirements

Firmware version on both HP 3PAR StoreServ arrays must be 3.2.1 or newer.

The sites must be set up in a Remote Copy 1-to-1 configuration with synchronous mode.

The round-trip latency between the two sites must be 2.6 milliseconds or less.

Quorum Witness software must be installed on a virtual machine at a third site.

The HP 3PAR StoreServ arrays communicate with Quorum Witness via the management port over TCP/IP. Quorum Witness needs to be reachable from both sites.

This non-disruptive failover configuration is supported with Windows Server 2008 R2, and Windows Server 2012 R2.

All source and target volumes exported from the two HP 3PAR StoreServ arrays must have the same volume Worldwide Name (WWN).

Hosts have to be connected to the HP 3PAR StoreServ Storage system using FC, iSCSI, or FCoE.

All associated hosts are connected to both arrays.

The Windows hosts that these volumes are exported to must be configured with host persona 15. It supports host capability ALUA configuration.

Both auto_failover and path_management polices are required to enable automatic transparent failover.

For the latest supported configuration and requirement for HP 3PAR Peer Persistence in a vSphere environment, refer to the Single Point of Connectivity Knowledge (SPOCK).

Host persona 15

Host personas are a set of behaviors that permit hosts connected to FC or iSCSI ports on the system to deviate from the default host behavior. By assigning a persona to a host, multiple host types that require distinct customized responses can share a single physical port on HP 3PAR StoreServ. For example, hosts running Windows, Linux, and AIX operating systems can all connect to the same HP 3PAR StoreServ port. This simplifies connecting hosts to the system and reduces management costs related to complex host connections.

In all versions since HP 3PAR OS 3.1.3, HP requires host persona 15 for Windows Server 2008, 2008 R2, 2012, or 2012 R2 hosts. Persona 15 supports ALUA path management and it must be used for host configurations that have Windows 2008 R2 and Windows 2012 R2 hosts connected to both the primary and the secondary arrays in an HP 3PAR Peer Persistence configuration. Existing Windows hosts defined with host persona other than host persona 15, such as persona 2, must be changed to persona 15. Refer to the HP 3PAR Windows Server 2012 and Windows 2008 implementation guide for more details about host creation and persona changes for those platforms.


5

Verify each VV has same WWN on both arrays

It is important that the source and the target volumes on the HP 3PAR StoreServ arrays have the same identity when both volumes are exported to the same Windows host. Hence, both volumes should have the same WWN. User can create volumes with the same WWN automatically using the new option in admitrcopyvv command. See the HP 3PAR StoreServ

Remote Copy software users guide at HP Storage Library for instructions on how to use the admitrcopyvv command. The HP 3PAR Management Console (MC) also creates VVs with the same WWN when configuring Remote Copy groups.

The replicated volume WWN can also be changed later if the volumes were created using the Management Console or any method other than the admitrcopyvv. See the HP 3PAR StoreServ Command Line Interface administrators manual at HP Storage Library or the HP 3PAR StoreServ Management Console online help for instructions on how to edit a VVs WWN.

Verify VLUN status on both arrays

Make sure that volumes are exported to the host from both HP 3PAR StoreServ arrays. After the volumes are exported from both arrays, the showvlun command will show the state as active for VLUNs exported from the source side and standby for the VLUNs exported from the target side. Windows hosts will send I/O requests only to the VLUNs reported as active.

Quorum Witness

The HP 3PAR StoreServ Quorum Witness (QW) software enables transparent automatic failover between the two sites in a Microsoft multisite cluster configuration. The Quorum Witness software gets regularly updated with status information from the HP 3PAR StoreServ arrays. In the event of a failure of one of the HP 3PAR StoreServ arrays, the surviving HP 3PAR StoreServ array detects that Quorum Witness is not getting updated by the failed array and initiates a failover operation on any secondary groups associated with that failed array.

Note QW details have to be set up at the target level. The automatic transparent failover will occur only for Remote Copy groups that use the target and have the auto_failover policy set. Failover is not automatic for Remote Copy groups between the same two HP 3PAR StoreServ systems that do not have the auto_failover policy set. The setting of path management is required as well. A volume group must have the path management policy set if it is to be valid for automatic and manual transparent failover. If these policies of auto_failover and path_management are set, then no automatic failover can occur if a disaster strikes.

Quorum Witness is set up on a third site where events that may impact site 1 or site 2 cannot impact the Quorum Witness site at the same time. Additionally, QW connects to arrays on both sites using non-RC links. With the above two configuration characteristics (site and link independence), QW helps determine the following nature of failures:

RC link failure: QW can detect if the two arrays are alive but not communicating because of an RC link failure. QW would still be receiving updates from both of the arrays.

Array or site failure: QW can detect when one of the arrays or sites fails. QW would not be receiving updates from one of the arrays that has failed.

The Handling failures section later in this paper covers the various failure conditions handled by Peer Persistence.

The Quorum Witness software is packaged as an Open Virtualization Format (OVF) package for deployment in a vSphere environment or a Hyper-V environment. The Quorum Witness virtual machine should not be stored on allocated storage from either of the arrays that are part of the Peer Persistence configuration. If the volume is located on one of the arrays that is part of the Peer Persistence configuration, failure of the array will make the volume unavailable, which will result in the loss of access to the Quorum Witness virtual machine.

Quorum Witness is a passive component of the configuration and the software itself does not provide any high availability. However, the virtual machine can be made highly available by using VMware High Availability or Fault Tolerance, or Microsoft Failover Clustering for the Hyper-V based QW package. For the latest supported configuration and requirement for HP 3PAR Peer Persistence QW, refer to the Single Point of Connectivity Knowledge (SPOCK).

HP 3PAR StoreServ systems communicate with Quorum Witness using the HP 3PAR StoreServ management interface. The network should be set up such that the QW server has network connectivity to the admin interface on both arrays. It is required to connect to the administration port from a minimum of two controller nodes to the network. However, if the array has more than two controllers, the best practice is to connect all controller nodes to the network. The Quorum Witness software uses port 8080 to communicate with the two HP 3PAR StoreServ systems in the quorum. Hence, firewall rules need to be changed to open port 8080 at the QW site.


6

Planned switchover

A switchover operation migrates the Remote Copy group role from source to target without impacting host I/O. This operation requires that the associated hosts are connected to the arrays on both sites. This is a planned operation and the Remote Copy groups need to already be started and synced for this command to work. This operation is useful when users are looking for workload mobility between the two sites. You can use the switchover command to reverse the replication,

resulting in site 2 hosts being able to access the logical unit numbers (LUNs) locally from site 2 array. The system performs the following action as part of a switchover operation:

I/O from the hosts to the volumes in the source Remote Copy group is blocked and in-flight I/Os are allowed to drain. The Remote Copy group is stopped and snapshots are taken on the primary array.

The primary array target port group is changed to transition state and it sends a remote failover request to the secondary Remote Copy group.

The secondary array target port group is then changed to transition state and it takes a recovery point snapshot.

The secondary Remote Copy group changes state to become primary-reversed and makes the volumes read/write.

The primary-rev target port group is changed to active state and the array returns a failover complete message to the primary Remote Copy group.

The primary array target port group is changed to standby state and any blocked I/O is returned to the host with a sense error: NOT READY, LOGICAL UNIT NOT ACCESSIBLE, TARGET PORT IN STANDBY STATE.

The host will then perform SCSI inquiry requests to detect what target port groups have changed and which paths are now active, and I/O will now be serviced on the active path to the primary-reverse Remote Copy group.

All operations to this point should complete within 30 seconds to ensure that host I/O does not timeout.

The primary Remote Copy group will then send a remote recover request to the primary-reverse Remote Copy group. The primary-reverse Remote Copy group will perform a recover operation and will change the primary Remote Copy group state to secondary-reverse state. The Remote Copy group will then be started from primary-reverse to secondary-reverse.

When the primary-reverse and secondary-reverse volumes are back in sync, the snapshots that were taken on both sides will be removed. The Remote Copy group will then undergo a reverse (-natural) operation. This operation will change the primary-reverse group to primary and secondary-reverse group to secondary.

HP 3PAR Peer Persistence failures handling

Peer Persistence with Quorum Witness can identify different types of failures and perform transparent failover automatically. HP 3PAR Peer Persistence will perform automatic transparent failover only for Remote Copy groups that have the auto_failover and path_management enabled.

Array-to-array communication failure

In the case of an array-to-array communication failure, HP 3PAR StoreServ will continue to send and receive communication to and from Quorum Witness. Automatic transparent failover does not occur and host I/Os continue to go to their primary volumes; however, replication of I/O across RC links will stop due to the failure.

Single site to QW communication failure

In the case where the arrays at either site 1 or site 2 lose connectivity with Quorum Witness, an automatic failover will not occur. Host I/O will continue and replication of I/O across RC links will continue normally. An automatic failover does not need to occur because the two HP 3PAR StoreServ arrays can still communicate with one another via the replication links they share.

Site 1 to QW and array-to-array communication failure

If the array in site 1 becomes isolated due to a dual network failure (communication with QW fails and the replication links to the array in site 2 fails simultaneously), Remote Copy groups that are in primary state on the array in site 1 will be transitioned to failsafe mode and will stop serving I/O to the ESXi hosts in site 1. The array in site 2 (which is still communicating with QW) will perform an automatic failover and host I/O will be redirected to the new primary volumes in site 2.

Site 2 to QW and array-to-array communication failure

This is the same as the previous case with the difference that site 2 will become isolated due to a dual network failure.


7

Site 1 and site 2 both lose communications to QW

In the case of a communication failure with QW from both sites, but where the Remote Copy links remain operational, both arrays will be aware of each others operational state. Automatic failover does not happen and host I/Os continue to the primary volumes; replication of I/O across RC links will continue as normal.

Site 1 and site 2 to QW and array-to-array communication failure

In the case where all network communication between the arrays and QW, and between both arrays fails, both arrays will be isolated as they can neither communicate with each other over the RC links nor with QW. In this case, Remote Copy groups that are primary will go into failsafe mode and stop serving I/O, and failover actions will not be performed. This will result in host I/O failure, and replication of I/O across RC links will stop due to the failure.

Table 1. HP 3PAR Peer Persistence failures handling

Replication stopped Automatic failover Host I/O impact

Array-to-array Remote Copy links failure Yes No No

Single site to QW network failure No No No

Single site to QW network and array-to-array Remote Copy link failure

Yes Yes No

Both sites to QW network failure No No No

Both sites to QW network and array-to-array Remote Copy link failure

Yes No Yes

Multisite Windows Server Failover Cluster and HP 3PAR Peer Persistence considerations

The implementation of the Microsoft multisite clustering requires certain considerations for optimal deployment. In the next section, we will discuss a few recommendations that the user should consider when deploying HP 3PAR Peer Persistence.

WSFC quorum model

The quorum configuration in a failover cluster determines the number of failures that the cluster can tolerate. The quorum voting method is determined by the quorum mode configured for the WSFC. These are the quorum modes available to use in WSFC:

Node majority: More than one-half of the voting nodes in the cluster must vote affirmatively for the cluster to be healthy.

Node and file share majority: It is similar to node majority quorum mode, except that a remote file share is also

configured as a voting witness, and connectivity from any node to that share is also counted as an affirmative vote. More than one-half of the possible votes must be affirmative for the cluster to be healthy.

Node and disk majority: It is similar to node majority quorum mode, except that a shared disk cluster resource is also

designated as a voting witness, and connectivity from any node to that shared disk is also counted as an affirmative vote. More than one-half of the possible votes must be affirmative for the cluster to be healthy.

Disk only: A shared disk cluster resource is designated as a witness, and connectivity by any node to that shared disk is

counted as an affirmative vote.

For multisite failover clustering with an odd number of nodes, the node majority can be used as quorum mode. However, in this configuration, if the site with more nodes fails, manual intervention would be required to force the cluster to start at the other site, because of loss of the quorum. With an even number of nodes, the quorum mode of node and file share majority is a recommended option. A file share witness (FSW) is recommended since it gives the user the option to place it in a third site. Placing an FSW in a VM on a third site with the HP 3PAR QW VM is an effective option for more resiliency with such deployment scenarios. The file share witness should not reside on any node or VM in the cluster, and it should be visible to all nodes in the cluster.


8

Cluster validation test

The cluster validation test is integrated in Failover Cluster Manager to run a set of validation tests. The validation will test the configuration and provide an assessment of how well failover clustering can be supported in a given configuration. Microsoft requires a cluster validation report as the condition for providing Microsoft support to a given cluster configuration.

Microsoft does acknowledge that in certain multisite cluster environments with SAN-based replication, the storage validation checks may fail or provide warnings. As a result, there are guidelines where the storage validation test can be skipped for the cluster validation tests. For more information, visit support.microsoft.com/kb/943984 and support.microsoft.com/kb/2914974.

For the HP 3PAR Peer Persistence deployment in a multisite Windows cluster, the cluster validation must be completed with all the tests. This will allow the user to address any issues in the configuration that might impact the user cluster environment later in production. Any failure or warning in the report should be addressed and carefully reviewed.

Networking consideration

Prior to the release of Windows Server 2008, the options for deploying failover clusters in multisite configurations were limited. Multisite failover clusters were restricted to certain environments because of the requirement that all the cluster nodes be on the same subnet.

Windows Server 2008 and Windows 2012 provide more flexibility in implementing multisite failover clusters. The requirement for stretched virtual local area networks (VLANs) to accommodate geographically dispersed hosts that are on different subnets is no longer needed. The failover cluster nodes can be deployed on different subnets across routed networks, and as a result eliminate the need to use VLANs.

Tuning of cluster heartbeat settings

In WSFC, the heartbeat is the process in which the nodes in the cluster can signal to one another that they are working. Since the release of Microsoft Windows 2008, it became possible for the user to tune the heartbeat settings in WSFC. As a result, there is more flexibility in certain deployments of multisite clusters in high-latency networks.

The setting to tune depends whether the nodes are on the same subnet or VLAN or if it is a multi-subnet deployment. By default, the threshold that defines the number of heartbeats that are missed before a recovery task occurs is five heartbeats for nodes located in the same subnet or different subnets. For the delay parameters, specify the amount of time that the cluster network driver waits between sending cluster service heartbeats.

Users should consider using the default values if possible and only change those values if there is the need to do so.

For information about tuning heartbeat settings in a multisite failover cluster, see go.microsoft.com/fwlink/?LinkId=130588.

Microsoft MPIO DSM

HP 3PAR uses the in-box Microsoft MPIO DSM for multipathing. For information about the installation and configuration of the MPIO, refer to the HP 3PAR Windows Server 2012 and Windows Server 2008 Implementation Guide.

When deploying HP 3PAR Peer Persistence for Windows multisite cluster, it is recommended to enable the path verification for the HP 3PAR disks on the Windows host (figure 17).

HP 3PAR Quorum Witness hosting

Given how critical QW is to the operation of HP 3PAR Peer Persistence, it is highly recommended that the QW VM be hosted outside the multisite cluster environment and should not be on the HP 3PAR arrays used in the HP 3PAR Peer Persistence deployment it arbitrates or a Windows host that is part of the multisite cluster.

It is possible to use a single QW to support up to four HP 3PAR Peer Persistence deployments. This will give users the option for consolidation to save on resources and for ease of management as well. For the latest supported configuration and requirement for HP 3PAR Peer Persistence QW, refer to the Single Point of Connectivity Knowledge (SPOCK).

Remote Copy replication links

The current release of HP 3PAR Peer Persistence with 3.2.1 supports FC, FCoE, and iSCSI host connectivity. However, it is recommended that Remote Copy of Fibre Channel (RCFC) be used for the synchronous replication of data between the arrays in a Peer Persistence configuration. Use of RCFC will enable a high-bandwidth and low-latency connection for the replication of data between the arrays, as it uses a patented protocol for synchronous replication that only requires a single round trip across the replication links to replicate an I/O smaller than 80K in size.


9

HP 3PAR Peer Persistence maximums

It is highly recommended to consolidate VVs inside a few RC groups in an HP 3PAR Peer Persistence configuration. Such a configuration will make sure the host I/Os will be failed over within the timing window in case of a transparent failover.

For the latest supported configuration, maximums, and requirement for HP 3PAR Peer Persistence, refer to the Single Point of Connectivity Knowledge (SPOCK).

Sample Peer Persistence configuration

This section gives the steps needed to set up a single Remote Copy group in Peer Persistence configuration. Refer to the HP 3PAR StoreServ Remote Copy software users guide for more details and the complete set of options available for the commands used in this section.

In this example, we are deploying the HP 3PAR Peer Persistence between two sites: Kirkland and Redmond. The multisite Windows Server Failover Cluster consists of three hosts running Windows Server 2012 R2.

Deploying Quorum Witness

The Quorum Witness software provides the arbitrator functionality in the Peer Persistence setup. The detailed instructions for deploying Quorum Witness are listed in the HP 3PAR StoreServ Remote Copy software users guide at the HP Storage Library. Refer to the users guide for more detailed information.

Requirements of Quorum Witness

HP 3PAR StoreServ Quorum Witness deployment requires:

Microsoft Hyper-V Server 2012 R2

2 GB memory

Network interface with access to a network with the HP 3PAR StoreServ operating system

IP address (Static is recommended)

QW installation

Refer to the RC user guide for further information on how to import and deploy QW on the supported hypervisor.

Note down the IP address displayed in the Summary tab after powering on the virtual machine. This IP address will be used when configuring Quorum Witness later.

Figure 2. HP 3PAR Peer Persistence QW VM


10

Creating Remote Copy targets

This is a software configuration guide only and assumes that the right hardware requirements for setting up Remote Copy are met. This guide will not cover how to configure ports for Remote Copy, how to do cabling, how to perform zoning, and so on. This guide also assumes that you have met all requirements for Peer Persistence.

Before you start replicating volumes from one array to another, you need to create Remote Copy targets on both arrays. Connect to both the arrays using the MC, then start with creating New Configuration in the Remote Copy Manager.

Select 1-1 configuration and select the two systems that will be used in the RC setup.

Figure 3. HP 3PAR RC configurationTarget

ConfigurationLinks Select the ports that will be used for Remote Copy by dragging and dropping the links between the ports. RCFC ports are

used in this example.

Click on finish as we would like to create groups later.


11

Figure 4. HP 3PAR RC Links configuration

Figure 5. HP 3PAR Remote Copy configuration


12

Peer Persistence configuration

Click on the Peer Persistence configuration tab after selecting the Remote Copy configuration navigation tree node. Use the IP address assigned to the QW during its configuration.

Do not click on the Start Witness checkbox at time of creating the configuration. Press OK after specifying the IP address.

Figure 6. HP 3PAR Peer Persistence configuration

After a brief moment the Quorum Status will change to Not-started. Select both targets in the list and click on the Start button to start the Quorum Witness.

Figure 7. HP 3PAR Peer Persistence configurationStart

The status should now change to Started.


13

Figure 8. HP 3PAR Peer Persistence configurationStatus

Redmond3PAR cli% showrcopy

Remote Copy System Information

Status: Started, Normal

Target Information

Name ID Type Status Options Policy

Kirkland3PAR 1 FC ready 2FF70002AC018575 mirror_config

Link Information

Target Node Address Status Options

Kirkland3PAR 0:2:3 20230002AC018575 Up


receive 0:2:3 20230002AC018575 Up

receive 1:2:4 21240002AC018575 Up

Troubleshooting Witness Quorum Communication issues

This operation needs to be done using CLI, so login to one of the HP 3PAR StoreServ arrays and perform the following:

Find the target name using showrcopy command.




Target Information



Link Information




14


receive 0:2:3 20230002AC018575 Up

receive 1:2:4 21240002AC018575 Up

Redmond3PAR cli% showrcopy -qw targets



Target Information

Name ID Type Status Policy QW-IP Q-Status Q-Status-Qual

Kirkland3PAR 1 FC ready mirror_config 192.168.0.34 Started

Use the witness check command to verify the communication between Quorum Witness and management interface of the two arrays. A basic message is displayed if the verification is successful.

Redmond3PAR cli% setrcopytarget witness check 192.168.0.34

Connectivity check passed

Redmond3PAR cli% setrcopytarget witness check -remote 192.168.0.34 Kirkland3PAR

Connectivity check passed

The following is an example if the Quorum Witness is not reachable:

Redmond3PAR cli% setrcopytarget witness check 192.168.0.37

error: No route to Quorum Witness at 192.168.0.37

Creating Remote Copy groups

Start the Create Remote Copy group wizard and select the source and target system. Next specify the group name and select synchronous mode. On the next screen, select the source volume you want to replicate and select the Create new volume on the target system. Its preferred to create new volumes on the target, this way the volumes will have the same characteristics as the source volumes. If you manually created volume beforehand, then be sure that the volume size on target matches the volume size on the source and both volumes have the same WWN (Figure 10).

Add all volumes in the group and click Finish. The group will start and initial synchronization will be started. More groups should be created. A typical configuration will have bi-directional HP 3PAR Remote Copy groups replication.

Figure 9. HP 3PAR Peer Persistence RC group creationVVs configuration


15

Figure 10. HP 3PAR Peer Persistence RC group creationVVs configuration

To continue with the configuration and the setup of the RC group, the user would need to add the VVs into RC groups and use synchronous replication.

Changing Group Policy

Its required to edit the replication group and enable auto_failover and Path_management policies to enable automat ic transparent failover.

Click on the Enable/Disable Groups button in the Peer Persistence tab. Select the group that you want to enable automatic failover and press OK ( Figure 11).

Figure 11. HP 3PAR Peer Persistence Auto Failover


16

This will enable the auto_failover and the path_management polices on the RC group.

Use the showrcopy command to verify the auto failover and path management settings.




Target Information



Link Information




receive 0:2:3 20230002AC018575 Up

receive 1:2:4 21240002AC018575 Up

Group Information

Name Target Status Role Mode Options

rc-gp-red-1 Kirkland3PAR Started Primary Sync

auto_failover,path_management

LocalVV ID RemoteVV ID SyncStatus LastSyncTime

clus1-red-csv-1 142 clus1-kir-csv-1 98 Synced NA








From the other array on the remote site, the same command would yield the following:

Kirkland3PAR cli% showrcopy



Target Information


Redmond3PAR 1 FC ready 2FF70002AC018576 mirror_config


17

Link Information


Redmond3PAR 0:2:3 20230002AC018576 Up

Redmond3PAR 1:2:4 21240002AC018576 Up

receive 0:2:3 20230002AC018576 Up

receive 1:2:4 21240002AC018576 Up

Group Information


rc-gp-red-1.r99702 Redmond3PAR Started Secondary Sync



clus1-kir-csv-1 98 clus1-red-csv-1 142 Synced NA








Exporting LUNs to hosts

When creating Windows hosts they need to be created using Persona 15. For best practice add all hosts in the cluster into a host set and add all VVs in a Remote Copy group into a VV set.

Figure 12. HP 3PAR Peer Persistence VV export to hosts in the Windows cluster


18

Figure 13. HP 3PAR Peer Persistence RC groups

Figure 14. Windows Server disk management view of the presented VVs


19

Figure 15. Windows Server MPIO state for HP 3PAR Peer Persistence VVs

Figure 16. Windows Server MPIO properties for HP 3PAR Peer Persistence VVs


20

Figure 17. Enable path verification

Figure 18. Windows cluster validation


21

Figure 19. Configuring cluster disk resources

Figure 20. Adding CSVs


22

Figure 21. Using the CSVs to run VMs

Planned switchover example

The CLI command has to be used to initiate a planned failover/switchover. The command has to be issued on the array with the primary role. This section provides details about the workings of this operation.

In this example the Remote Copy groups are primary on Redmond3PAR.




Target Information



Link Information




receive 0:2:3 20230002AC018575 Up

receive 1:2:4 21240002AC018575 Up

Group Information








23







The Remote Copy groups are secondary on Kirkland3PAR.




Target Information



Link Information


Redmond3PAR 0:2:3 20230002AC018576 Up

Redmond3PAR 1:2:4 21240002AC018576 Up

receive 0:2:3 20230002AC018576 Up

receive 1:2:4 21240002AC018576 Up

Group Information














24

Figure 22. The host path status before the switchover

Figure 23. HP 3PAR Peer Persistence switchover using the HP 3PAR MC


25

After the switchover command is executed successfully the primary role will move to Kirkland3PAR




Target Information



Link Information




receive 0:2:3 20230002AC018575 Up

receive 1:2:4 21240002AC018575 Up

Group Information


rc-gp-red-1 Kirkland3PAR Started Secondary Sync














Target Information




26

Link Information


Redmond3PAR 0:2:3 20230002AC018576 Up

Redmond3PAR 1:2:4 21240002AC018576 Up

receive 0:2:3 20230002AC018576 Up

receive 1:2:4 21240002AC018576 Up

Group Information


rc-gp-red-1.r99702 Redmond3PAR Started Primary Sync











Figure 24. Host path status change after the switchover


27

Figure 25. Host I/O load switch between the two arrays during a switchover

Recovering from automatic failover

If one array fails, then primary Remote Copy groups from the array that has auto_failover policy enabled will be failed over to the secondary array. Recovery process is needed to restart replication and reverse replication back in the original direction.




Target Information



Link Information




receive 0:2:3 20230002AC018575 Up

receive 1:2:4 21240002AC018575 Up

Group Information








28










Target Information



Link Information


Redmond3PAR 0:2:3 20230002AC018576 Up

Redmond3PAR 1:2:4 21240002AC018576 Up

receive 0:2:3 20230002AC018576 Up

receive 1:2:4 21240002AC018576 Up

Group Information













After the Redmond3PAR become inaccessible, the Kirkland3PAR become the primary-rev. The RC links are down and the replication is stopped.




Target Information


Redmond3PAR 1 FC failed 2FF70002AC018576 mirror_config


29

Link Information


Redmond3PAR 0:2:3 20230002AC018576 Down

Redmond3PAR 1:2:4 21240002AC018576 Down

receive 0:2:3 20230002AC018576 Up

receive 1:2:4 21240002AC018576 Up

Group Information


rc-gp-red-1.r99702 Redmond3PAR Stopped Primary-Rev Sync



clus1-kir-csv-1 98 clus1-red-csv-1 142 Stopped 2014-10-15 15:11:16 PDT



rc-gp-red-2.r99702 Redmond3PAR Stopped Primary-Rev Sync





After the array is recovered and back online the showrcopy status will show




Target Information



Link Information




receive 0:2:3 20230002AC018575 Up

receive 1:2:4 21240002AC018575 Up

Group Information


rc-gp-red-1 Kirkland3PAR Failsafe Primary Sync



clus1-red-csv-1 142 clus1-kir-csv-1 98 Failsafe NA



30


rc-gp-red-2 Kirkland3PAR Failsafe Primary Sync





These steps should be used to recover the Remote Copy group after the array that had failed is back online.

Select the group(s) that needs to be recovered under the Remote Copy configuration node. Click on the Recover button and press OK. Select Yes at the next prompt to continue with the recovery.

Figure 26. HP 3PAR Peer Persistence recover

The Restore button will be enabled after the recovery has completed. However, performing the restore operation might lead to the Windows hosts losing access to the LUNs for a short duration. Hence its recommended to use the CLI commands to complete the failback.

Note

It is also required that a manual device rescan is performed on each node in the Windows Server Failover Cluster in order to work around an issue with the MPIO DSM not detecting the change of status of the restored paths to the previously inaccessible array.

Login to the array with the primary-rev role and issue the following command.

Kirkland3PAR cli% setrcopygroup reverse -f -natural rc-gp-red-1.r99702 rc-gp-red-

2.r99702

setrcopygroup for reverse -natural rc-gp-red-1.r99702 rc-gp-red-2.r99702

reverse started with tasks: 4602 4603


31

Now the two array would show the following;




Target Information



Link Information


Redmond3PAR 0:2:3 20230002AC018576 Up

Redmond3PAR 1:2:4 21240002AC018576 Up

receive 0:2:3 20230002AC018576 Up

receive 1:2:4 21240002AC018576 Up

Group Information
















Target Information



Link Information




receive 0:2:3 20230002AC018575 Up

receive 1:2:4 21240002AC018575 Up


32

Group Information













In order to restore the primary and secondary roles back to their state prior to the failure, a switchover is needed.

Kirkland3PAR cli% setrcopygroup switchover rc-gp-red-1.r99702 rc-gp-red-2.r99702

This command will migrate the Remote Copy group(s) without impacting the host I/O.

The direction of the indicated group(s) will be switched on both the local and the

group's target system.

Are you sure that you wish to perform this action?

select q=quit y=yes n=no: y

setrcopygroup for switchover rc-gp-red-1.r99702 rc-gp-red-2.r99702

switchover started with tasks: 4604 4605

This will bring the RC group back to their status prior to the failure.




Target Information



Link Information


Redmond3PAR 0:2:3 20230002AC018576 Up

Redmond3PAR 1:2:4 21240002AC018576 Up

receive 0:2:3 20230002AC018576 Up

receive 1:2:4 21240002AC018576 Up


33

Group Information
















Target Information



Link Information




receive 0:2:3 20230002AC018575 Up

receive 1:2:4 21240002AC018575 Up

Group Information














Sign up for updates

hp.com/go/getupdated

Share with colleagues

Rate this document

Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for

HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as

constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Microsoft, Windows, and Windows Server are trademarks of the Microsoft Group of companies. Linux is the registered trademark of Linus Torvalds in the

U.S. and other countries. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions.

4AA5-5894ENW, November 2014

For more information

HP Enterprise Storage Information Library, hp.com/Go/Storage/Docs

HP 3PAR StoreServ Storage: hp.com/go/storeserv

HP Support Center, HP 3PAR StoreServ Implementation Guide

HP 3PAR StoreServ Storage best practice guide 4AA4-4524ENW

HP 3PAR Remote Copy Software User Guide OS 3.2.1

Learn more at hp.com/solutions/activeanswers

implementing microsoft multisite clustering using hp 3par peer persistence

Documents

array communication

qw communication failure

single site

microsoft multisite

quorum witness

wsfc quorum model

technical white paper

vlun status