high availability configuration for healthcare … · high availability configuration for...

White Paper

ECD Healthcare

Abstract

This white paper describes a solution that offers high availability for EMC Documentum xDB document sharing as part of the Healthcare Integration Portfolio product suite. The paper also describes how to determine the functionality of the components within the virtualized solution.

July 2015

HIGH AVAILABILITY CONFIGURATION FOR HEALTHCARE INTEGRATION PORTFOLIO (HIP) REGISTRY EMC Documentum HIP, EMC Documentum xDB, Microsoft Windows 2012

• High availability for EMC Documentum xDB • Automated recovery

High Availability Configuration for Healthcare Integration Portfolio (HIP) Registry EMC Documentum HIP, EMC Documentum xDB, Microsoft Windows 2012

2

Copyright © 2015 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

All information included in this document is provided "as-is" and does not guarantee EMC's support or approval. Please consult the official product documentation or the Product Support team regarding any questions of supportability.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All trademarks used herein are the property of their respective owners.

Part Number H14229

http://www.emc.com/legal/emc-corporation-trademarks.htm

http://www.emc.com/legal/emc-corporation-trademarks.htm

3 High Availability Configuration for Healthcare Integration Portfolio (HIP) Registry

EMC Documentum HIP, EMC Documentum xDB, Microsoft Windows 2012

Table of contents

Executive summary ............................................................................................................................... 5 Business case .................................................................................................................................. 5 Solution overview ............................................................................................................................ 6 Key results/ recommendations ........................................................................................................ 6

Introduction.......................................................................................................................................... 7 Purpose ........................................................................................................................................... 7 Scope .............................................................................................................................................. 7 Audience ......................................................................................................................................... 7 Terminology ..................................................................................................................................... 7

Technology overview ............................................................................................................................ 8 Overview .......................................................................................................................................... 8 EMC Documentum HIP components ................................................................................................. 8 EMC Documentum xDB .................................................................................................................... 8 VMware vSphere .............................................................................................................................. 8 Microsoft Server Failover Clustering ................................................................................................. 9

Architectural overview ........................................................................................................................ 10 Physical environment ..................................................................................................................... 10 Hardware resources ....................................................................................................................... 11 Software resources ........................................................................................................................ 11 VMware datastores ........................................................................................................................ 11

Virtual configuration .......................................................................................................................... 13 ESX configuration ........................................................................................................................... 13 VMware network ............................................................................................................................ 13 Virtual machine resources and software ......................................................................................... 13

Design considerations ........................................................................................................................ 15 DNS configuration .......................................................................................................................... 15 Microsoft Server Failover Clustering ............................................................................................... 15

xDB Master (Read Write) Cluster configuration .......................................................................... 16 xDB Replica (Read Only) Cluster configuration ........................................................................... 19

Test results ........................................................................................................................................ 23 Overview ........................................................................................................................................ 23 Test objectives ............................................................................................................................... 23 Results and observations ............................................................................................................... 24

xDB Read Write master failover .................................................................................................. 24


4

xDB replica failover ................................................................................................................... 27

Conclusion ......................................................................................................................................... 29 Summary ....................................................................................................................................... 29 Findings ......................................................................................................................................... 29

References.......................................................................................................................................... 30 Product documentation .................................................................................................................. 30



Executive summary This white paper describes the high availability (HA) solution for the Healthcare Integration Portfolio (HIP) Registry using EMC Documentum® xDB, Microsoft Server Failover Clustering™, VMware® vSphere®, and Microsoft Windows Server.

The purpose of this white paper is to demonstrate the automated failover capability of EMC Documentum xDB using Microsoft Server Failover Clustering. This solution provides a cross-document sharing environment including:

• EMC Healthcare Integration Portfolio (HIP)

• EMC Documentum xDB

• Microsoft Server Failover Clustering

• VMware vCenter™

We1 built and tested this solution on a VMware virtualized infrastructure.

The architecture used for testing provides:

• Guidelines to build similar solutions

• A high availability procedure using Microsoft Failover Clustering

Healthcare IT management is challenged by the growing volumes of medical imaging data that result from:

• Technological advances in modalities

• Increased number of diagnostic studies and number of images per study

• Lengthening of diagnostic image retention requirements

Compounding these challenges is the fact that healthcare IT management also faces the difficulty of integrating and being able to easily share unstructured data with other patient data and presenting this data to the caregiver at the “point-of-decision-making” anywhere in the enterprise. Unstructured data includes medical images from departmental Picture Archiving and Communication Systems (PACS), such as radiology, cardiology, oncology, pathology, and mammography, for which healthcare organizations need to make informed decisions at the point of care.

To provide optimum patient-care delivery, IT managers are investing in automated tools to integrate the “stovepipes” of information created by numerous proprietary applications, and to efficiently manage the increasingly complex infrastructure of both physical and virtual assets in their environments.

To move toward digital-care processes, healthcare organizations typically move through multiple investment phases, such as clinical workflow processes. They must change in parallel with IT investments, including adoption of and adherence to industry standards as well as information digitization, sharing, and management.

1 The use of “we” refers to the team of EMC and partner solutions engineers who designed, built, and validated the solution.

Business case


6

To support these investments, healthcare providers need to address the following:

• Increasing amounts of storage required to support legacy and new healthcare applications

• Increasing costs and operational impact of maintaining different PACS infrastructures

• Adopting DICOM and HL7 standards that facilitate the sharing of images and critical information associated with them

• Providing a higher level of transparency and information access to patient data and medical images

• Centralizing, securing, and managing data storage

As healthcare systems focus more on collaborative patient care and the continuity of care delivered through a network of healthcare providers operating out of traditional settings, the need for flexible and cost-effective information systems is more critical. Your organization needs collaboration across various settings, but traditional IT infrastructures cannot always effectively support this due to cost issues, long development time, and a lack of flexibility.

This solution provides high availability for EMC Documentum xDB systems and shows the ability of Microsoft Failover Clustering to automate the failover and recovery processes.

This solution enables customers to:

• Provide high-availability for EMC Documentum xDB systems

• Automate the failover of the xDB Master and Replica Server to a standby Master and Replica Server using Failover Clustering

• Access clinical documents with read/write privileges in the event of a failure of the xDB database

The testing of this solution enables you to understand the requirements to create a high-availability solution for xDB. It provides you with the background to explain the recovery time and procedures needed for the environment to be automatically brought back online when the xDB servers experience an outage.

Solution overview

Key results/ recommendations



Introduction

The purposes of this white paper are to validate:

• The test methodology used to determine the functionality of the high-availability HIP solution

• xDB Server automated high availability using Microsoft Failover Clustering

The scope of this white paper includes:

• Measuring functionality of Microsoft Failover Clustering recovery technology

• Documenting the process of automated xDB master and replica recovery using Microsoft Failover Clustering

This white paper is intended for EMC employees, partners, and customers, including IT planners, xDB Server database administrators (DBAs), and EMC field personnel who are tasked with deploying such a solution in a customer environment. It is assumed that the reader is familiar with the various components of the solution.

This white paper includes the terminology shown in Table 1.

Table 1. Terminology

Term Definition

DICOM Digital Imaging and Communications in Medicine

HIP Healthcare Integration Portfolio

HL7 Health Level 7

IHE Integrating Healthcare Enterprise

PACS Picture Archiving and Communication System

RIS Radiology Information System

WDK Web Development Kit

xDB Documentum XML database

XDS Cross-Enterprise Document Sharing

Purpose

Scope

Audience

Terminology


8

Technology overview

This section provides an overview of the primary technologies included in this solution:

• EMC Documentum HIP components:

XDS Registry

XDS Repository

• EMC Documentum xDB

• VMware vSphere

• Microsoft Failover Clustering

The HIP XDS Registry is the directory or “white pages” for medical and administrative content that provides applications within the enterprise with the ability to discover and then access information. Based on the implementation of Integrating Healthcare Enterprise’s (IHE) XDS Registry specifications, it provides a central catalog for documents that reside in a federated system or in repositories, either heterogeneous or geographically distributed.

The HIP XDS Repository stores structured and unstructured healthcare information where all patient-centric documents, images, and media are available via the XDS-specification for applications to consume—even when the clinical, financial and operational content was not created via an XDS specification.

EMC Documentum xDB is a native XML database. Its scalable architecture and complete support for the XQuery language enables any organization to warehouse content in an application-neutral format. These databases do not depend on a proprietary application for information retrieval.

Unlike relational databases, xDB enables you to easily modify content schemas to adapt to changing information requirements and supports queries against complex data structures that are not easily modeled in rows and columns. In addition, with its powerful and extensible development and runtime toolset, xDB is a powerful platform for the most complex and demanding content-centric applications.

VMware vSphere uses the power of virtualization to transform data centers into simplified cloud computing infrastructures and enables IT organizations to deliver flexible and reliable IT services. vSphere virtualizes and aggregates the underlying physical hardware resources across multiple systems and provides virtual resource pools to the data center.

As a cloud operating system, vSphere manages large infrastructure collections (such as CPUs, storage, and networking) as a seamless and dynamic operating environment. It also manages the complex operations of a data center.

Overview

EMC Documentum HIP components

EMC Documentum xDB

VMware vSphere



Microsoft Server Failover Clustering monitors and automates the restart process of your application and data services on an alternate server for Microsoft Windows environments. Failover Clustering also automates the failback of your services, applications, and data—managing both failover directions and ensuring consistent and error-free business continuity.

Microsoft Server Failover Clustering


10

Architectural overview This white paper characterizes and validates high availability for a virtualized EMC Documentum xDB and XDS environment using Windows 2012 R2 and Microsoft Server Failover Clustering. VMware is used to provide the virtualization layer.

Figure 1 shows the physical environment.

Physical environment Figure 1.

Physical environment



Table 1 shows the hardware resources used in this solution.

Table 1. Hardware resources

Equipment Quantity Configuration

Server 2 128 GB memory servers with 2.27 GHz, 8C processors

VNX5500 storage array 1 • 75 x 600 GB SAS 10k drives

• 75 x 2 TB NL-SAS 7.2k drives

Fibre Channel (FC) switch 2 8 GB, 80 ports

Network switch 1 10 GB, 24 ports

Table 2 shows the software resources used in this solution.

Table 2. Software resources

Software Version Notes

EMC VNX5500

VNX OE for Block

FLARE™ release 32 Operating environment for the block

VMware ESX 5.1 Update 1 Server hypervisor

EMC PowerPath® Virtual Edition 5.9 Multipathing and load balancing for block access

VMware vCenter Server 5.1 Update 1 vSphere Management Server

Microsoft Windows Server 2012 R2 Operating system for all virtual machines

EMC Documentum xDB 10.5.9 Installed on Microsoft Windows. Includes xDB Master and xDB Replica

EMC Documentum HIP components 1.7 Installed on Microsoft Windows. Includes XDS Registry and Repository

EMC Documentum 7.2

Used for hosting Documentum Repositories

As shown in Table 3, we created a 70 GB VMFS datastore per virtual machine to house each of the virtual machines.

We also created an individual VMware Virtual Machine File System (VMFS) datastore for each of the databases and log disks.

Hardware resources

Software resources

VMware datastores


12

Table 3. VMWare Datastore configuration details

Virtual machine Size/GB Type Datastore Drive letter

xDB Master 70 VMFS xDB Master C:\

48 VMFS xDB database D:\

18 VMFS xDB Log L:\

xDB Master Standby

70 VMFS xDB Master Standby C:\

xDB Replica 70 VMFS xDB Replica C:\

48 VMFS xDB Replica database D:\

18 VMFS xDB Replica log L:\

xDB Replica Standby

70 VMFS xDB Replica Standby C:\

XDS Registry 70 VMFS XDS Registry C:\

XDS Repository 70 VMFS XDS Repository C:\

Documentum 70 VMFS DCTM node C:\

Documentum file store

200 VMFS Filestore1 E:\



Virtual configuration

The virtual infrastructure consisted of two servers to form a VMware ESX HA cluster with version 5.1 installed.

We configured all virtual machines to use VMXNET3 adapters. This included a 10 GbE backbone network connecting the physical ESX servers. We connected a virtual distributed switch for the ESX cluster into which the 10 Gb ports were added.

Table 4 lists the virtual resources allocated to each of the virtual machines and the software we installed on each machine.

Table 4. Virtual machine allocations

Virtual machine Quantity Software

Documentum Content Server and Docbroker

1 • Microsoft Windows Server 2012 R2

• Apache Tomcat 7

• Microsoft SQL Server 2012 Client

• Java JDK 1.7.0_67

• Documentum Content Server 7.2 containing:

XDS Repository

Content Storage Services enabled

Retention Policy Services enabled

Trusted Content Services enabled

Documentum Docbroker

Solution custom DAR file installed to XDS Repository

• VMware tools

Documentum XDS Registry


• Java JDK 1.7.0_67

• Documentum HIP Registry 1.7

• Tomcat 7

• VMware tools

XDS Repository 1 • Microsoft Windows Server 2012 R2

• Java JDK 1.7.0_67

• HIP Repository 1.7

• Tomcat 7

• VMware tools

ESX configuration

VMware network

Virtual machine resources and software


14

Virtual machine Quantity Software

xDB Master (Read Write) Server


• Java JDK 1.7.0_67

• Documentum xDB 10.5.9

• Tomcat 7

• VMware tools

• Microsoft Windows Server Failover Clustering

xDB Replica (Read Only) server


• Java JDK 1.7.0_67

• Documentum xDB 10.5.9

• Tomcat 7

• VMware tools

• Microsoft Windows Server Failover Clustering



Design considerations We designed the xDB Master Read Write Servers to share the VMFS database and log disks, as shown in Figure 2, so that the standby server could seamlessly take over the function of the Master Server in the event of a failure.

The xDB application was installed on the master node with the database and logs placed on the respective drives. We then installed xDB on the Master Standby Server.

We designed the xDB Read Only Servers in the same way with their own database and log disks.

xDB server disk configuration Figure 2.

We created a DNS alias for the xDB Master Server called W2K12xDBRWClus. This alias was then used for all communication between the Read Write Cluster and Read Only Clusters.

Microsoft Server Failover Clustering monitors and automates the restart process of your application and data services on an alternate server for Microsoft Windows environments. Failover Clustering also automates the failback of your services, applications, and data—managing both failover directions and ensuring consistent and error-free business continuity.

DNS configuration

Microsoft Server Failover Clustering


16

xDB Master (Read Write) Cluster configuration

After xDB has been installed on the master node(s) the EMC xDB service needs to be set to manual so that the Microsoft Failover Cluster Service can manage the service

The xDB Read Write Cluster was configured with the EMC xDB Server service as shown in Figure 3. The EMC xDB server role was configured using a generic service type. The configuration also contained the xDB master and xDB master standby virtual machines, as well as the following configuration settings:

xDB server service Figure 3.



Figure 4 shows the Node list in xDB master (Read Write) Microsoft Failover Cluster.

xDB master (Read Write) Cluster Node Configuration Figure 4.

Figure 5 shows the disks created in Microsoft Failover clustering.

xDB master (Read Write) Cluster disks Figure 5.


18

Figure 6 and Figure 7show the xDB master login screen and main screen prior to failure.

xDB Master (Read Write) login screen Figure 6.

xDB Master (Read Write) main screen Figure 7.



xDB Replica (Read Only) Cluster configuration

After xDB has been installed on the replica node(s) the EMC xDB service needs to be set to manual so that the Microsoft Failover cluster service can manage the service

Figure 8 shows the script that needs to be run on one of the replica nodes, in each cluster, once during its lifetime. This is run after the xDB software has been deployed on both cluster nodes. The reason why this script is required is because in xDB, a replica federation is created by copying a master federation. After starting a replica server on the replica federation, the replica server keeps in sync with the master by replaying transaction logs that come from the master. The master federation knows about the replica because a replica id has been registered.

The master server keeps obsolete log records it would otherwise delete, until the replica server ask for them and confirms it got them. One of the unwanted effects of creating a replica is that the replica on its turn has the same replica id registered and on its turn keeps otherwise obsolete log records until yet another replica would ask for its log records

xDB Read only create replica script Figure 8.

Figure 9 shows the run script that brings the xDB application online on the xDB Replica Read Only cluster. The create script above needs to be executed before the run script is configured in the cluster configuration. This ensures that the server is synchronized with the xDB master Read Write server. We connected to the xDB alias, W2K12xDBRWClus, to ensure all updates were synchronized to the replica.

xDB Read only run server script Figure 9.

call xdb.bat --debug -y create-replica --replicabootstrappath d:\\data\\XhiveDatabase.bootstrap --replicaid 10 --federation xhive://W2K12xDBRWClus:1235 --password xxxxxxx

start /B xdb.bat --debug -y run-server --master xhive://W2K12xDBRWClus:1235 --webserver-port 0 --address 0 --federation d:\\data\\XhiveDatabase.bootstrap --port 1235 --replicator 10

call xdb.bat --debug -y create-replica --remove 10 --federation xhive://W2K12xDBROClus:1235 --password xxxxxxx

call xdb.bat --debug -y stop-server --federation xhive://W2K12xDBROClus:1235 --password xxxxxxx

xdb run-server --master xhive://W2K12xDBRWClus:1235 --replicator 10 --address 0 --federation d:\\data\\XhiveDatabase.bootstrap


20

Figure 10 and Figure 11 show the configuration of the EMC xDB Server service within the xDB Replica (Read Only) Cluster. The xDB Read only role was configured as a generic application with the run server script shown in Figure 11. The configuration contained the xDB replica and xDB replica standby virtual machines as well as the disks

xDB Replica (Read Only) Cluster Configuration Figure 10.



xDB Replica (Read Only) Cluster Configuration Figure 11.

Figure 12 shows the node list in xDB Read Only Microsoft Failover Cluster.

xDB Replica (Read Only) Cluster Node Configuration Figure 12.


22

Figure 13 shows the disks that we used in xDB Read Only Microsoft Failover cluster.

xDB Replica (Read Only) Cluster disks Figure 13.

Figure 14 shows the xDB Read Only Replica Server login screen, with the W2K12xDBRWClus, replication master.

xDB Replica (Read Only) login screen Figure 14.



Test results

This chapter documents the results and observations made during the testing of high availability for Documentum HIP xDB.

The objectives of the test were to initiate a failure of the xDB Master Read Write and xDB Read Only Replica Server during clinical document ingestion and observe the recovery process.

To validate the environment we executed the ProvideandRegister Registry call to ingest clinical data into the XDS Registry.

All the testing was based on a single site implementation of EMC Documentum Healthcare Information Portfolio (HIP) and EMC Documentum xDB.

These tests did not include the failover of XDS Registry or XDS repository. For more information on the failover of these products refer to the following document: EMC Medical Image Management with Document Sharing Solution (Enterprise) – Business Continuance

Overview

Test objectives


24

xDB Read Write master failover

Figure 15 shows the Microsoft Failover interface during a failure of the xDB master node.

Microsoft Failover Interface showing xDB Master (Read Write) failover Figure 15.

Results and observations



Figure 16 shows the Microsoft Failover cluster Interface after a failover. You can see that the resources are now located on the xDB master standby node W2K12XDB-2. The total time to fail over from the master to the standby was approximately 15 seconds.

The Replica Server automatically reconnects to the Master Server for updates.

Microsoft Failover Interface showing xDB Master (Read Write) after failover Figure 16.


26

Figure 17and Figure 18 show the xDB master login screen and the main screen after failover.

xDB Master (Read Write) master login screen Figure 17.

xDB Master (Read Write) master main screen Figure 18.



xDB replica failover

Figure 19 shows the Microsoft Failover Cluster interface during a failure of the xDB replica node.

Microsoft Failover console during xDB Replica (Read only) failover Figure 19.

Figure 20 shows the xDB replica login screen after failover. The replica receives updates from the xDB Master Server W2K12xDBRWClus. The Replica Server automatically reconnects to the Master Server for updates.

The total time for failover of the replica is ten seconds.


28

xDB Replica (Read only) login screen after failover Figure 20.



Conclusion

The testing emulated failure scenarios within a Cross-Enterprise Document Sharing environment and shows that the solution provides the following benefits:

• High availability for EMC Documentum xDB using Microsoft Failover Clustering

• The use of standby servers for the xDB Read Write master and xDB Read Only replica provides resilience

The test results demonstrate that this solution provides:

• xDB master failover in approximately fifteen seconds

• xDB replica failover in approximately ten seconds

• Automatic reconnection of the xDB replica to the xDB master

Summary

Findings


30

References

For additional information, see EMC.com and the product documents listed below.

• Industry Solutions on EMC.com

• Choosing an xDB Configuration

• EMC Medical Image Management with Document Sharing Solution (Enterprise) – Business Continuance

Product documentation

http://www.emc.com/campaign/iig-healthcare/index.htm

https://community.emc.com/docs/DOC-3004

high availability configuration for healthcare … · high availability configuration for...

Documents