A S O L U T I O N S W H I T E P A P E R
from VERITAS Software Corporation
VERITAS volume replication technology, and how it enhances disaster tolerance in Oracle databases
VE
RIT
AS
Vol
ume
Rep
licat
ion
an
d O
racl
e D
atab
ases
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
3
Abstract Remote replication of online disks and volumes is emerging as the technique of
choice for protecting enterprise data against environmental disasters. Recognizing
this, VERITAS has embarked on aggressive campaign to build disaster recovery solu-
tions around the VERITAS Volume Replicator for major UNIX platforms.
Oracle Corporation, the leading supplier of relational database technology for the en-
terprise, recognizes the importance of disaster recoverability as well, and has intro-
duced its own replication technology that works cooperatively with the volume repli-
cation technologies of other vendors such as VERITAS. In addition, Oracle has insti-
tuted an audited testing program to verify that its technology interoperates correctly
with other vendors’ volume replication technologies for database disaster recovery.
VERITAS is proud to be the first vendor of volume replication technology to success-
fully complete the Oracle Storage Compatibility Program (OSCP) suite of disaster
recoverability tests.
This paper describes the capabilities of the VERITAS Volume Replicator, and how it
interoperates with Oracle’s standby database replication technology to provide robust
disaster recoverability for enterprise Oracle databases.
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
5
Contents I. ONLINE STORAGE FOR THE ENTERPRISE.................................................................................... 7
ENTERPRISE DATA STORAGE REQUIREMENTS ............................................................................................ 7 THE SOLUTION: RELATIONAL DATABASES AND LOGICAL VOLUMES ........................................................ 7 THE VERITAS VOLUME MANAGER .......................................................................................................... 8
II. DATA REPLICATION......................................................................................................................... 13 THE NATURE OF DATA REPLICATION ....................................................................................................... 13 THE VERITAS VOLUME REPLICATOR ..................................................................................................... 13 SYNCHRONOUS AND ASYNCHRONOUS REPLICATION................................................................................ 16
III. VOLUME REPLICATION AND ORACLE ..................................................................................... 23 ORACLE DATABASE REPLICATION TECHNIQUES ...................................................................................... 23 HYBRID REPLICATION OF ORACLE DATABASES ....................................................................................... 24 COMPARISON OF ORACLE REPLICATION TECHNIQUES.............................................................................. 25
IV. SUMMING UP ..................................................................................................................................... 29 VERITAS TOTAL DATA STORAGE MANAGEMENT .................................................................................. 29 WHY THE VERITAS SOLUTION? ............................................................................................................. 29
Figures FIGURE 1: REPLICATION WITH VERITAS VOLUME REPLICATOR ................................................................. 13 FIGURE 2: VERITAS VOLUME REPLICATOR DATA FLOW ............................................................................ 16 FIGURE 3: SYNCHRONOUS REPLICATION TIME LINE ..................................................................................... 17 FIGURE 4: APPLICATION PERFORMANCE WITH SYNCHRONOUS AND ASYNCHRONOUS REPLICATION............ 18 FIGURE 5: IN-BAND CONTROL MESSAGE TIME LINE..................................................................................... 21 FIGURE 6: RECIPROCAL DISASTER RECOVERY .............................................................................................. 22
Tables TABLE 1: VOLUME TYPES SUPPORTED BY THE VERITAS VOLUME MANAGER.............................................. 8 TABLE 2: COMPARISON OF STANDBY DATABASE HYBRID AND PURE VOLUME REPLICATION ...................... 26
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
7
I. Online Storage for the Enterprise
High-performance continuous access to online data
Enterprise Data Storage Requirements
The Requirement: Continuously Accessible, Trustworthy Data
Continuous access to data has become an operational necessity for enterprises. Cus-
tomers, suppliers, and even employees expect to be able to access enterprise informa-
tion resources all the time. Downtime, whether due to hardware or software failure,
the need to back data up, or because an upgrade is necessary, is no longer an option.
For managers and administrators responsible for information services data storage
management strategies, the challenges of continuous data availability are formidable:
➨ Maintaining the integrity of enterprise data while meeting increasingly com-plex, rapidly changing application needs.
➨ Providing continuous access to data, even if applications, systems, or entire data centers become incapacitated.
➨ Providing I/O performance that scales with enterprise requirements.
➨ Containing management cost as the amount of data increases.
The Solution: Relational Databases and Logical Volumes
Technology offers two complementary answers to enterprise needs for correct, con-
tinuously available, scalable, manageable data: relational database management sys-
tems and logical storage volumes.
For data integrity, relational database management systems such as Oracle provide:
➨ enterprise-wide application-independent data formats,
➨ data item integrity through constraints and user-definable triggered procedures,
➨ transactional integrity through logging and commitment/rollback techniques,
➨ high I/O performance, through intelligent use of massive cache memories,
➨ manageability through extensive management toolsets.
Relational database management systems do an outstanding job of keeping data error-
free and accessible at high performance levels. To do their job optimally, however,
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
8
they need storage they can rely on. Oracle is more robust, performs better, and is
more economical to own in the long run if the files or raw storage that hold its user
data and metadata are likewise robust, high performing, and flexibly manageable.
Logical volume management technology aggregates disks, and presents the database
management system with online storage units that are functionally disk-like, but have
superior availability, performance, and manageability. Software packages such as the
VERITAS Volume Manager provide logical volume management. RAID controllers
also provide a level of logical volume management. Wherever it runs, volume man-
agement software has two basic responsibilities:
➨ It protects data against loss or loss of access due to disk or channel hardware failures. Mirroring and RAID are the two techniques commonly employed.
➨ It maps disk-like volume block address spaces to disk blocks for optimal data access performance. Data striping is the most common mapping technique.
The VERITAS Volume Manager
The VERITAS Volume Manager runs as an operating system layer between I/O driv-
ers and the file system or database management system, VERITAS Volume Manager
enhances storage availability, performance, and manageability by presenting parts of
disks, entire disks, or groups of disks as logical volumes. Table 1 indicates the types
of volumes supported by the VERITAS Volume Manager in Solaris, HP-UX, and
Windows server operating system environments.
Volume Type Disk Failure Tolerance
(Data Availability)
Performance (relative to single disk)
Enterprise IT Application
Mirrored (two or more copies)
(“RAID 1”)
Very high High read performance Comparable write performance
Small (single-disk) critical files
Striped and Mirrored
(n-way)
Very high Very high read performance High write performance
Large (multi-disk) critical files
Striped with Parity
(“RAID 5”)
High High read performance Low write performance
Important “read-mostly” data
Striped without Parity
(“RAID 0”)
Lower than single disk High read and write performance
Limited to easily replaceable performance critical data
Concatenated (capacity
aggregated)
Lower than single disk Comparable read and write performance
Limited to easily replaceable non-performance critical data
Simple Same as single disk Same as single disk Small, easily replaceable non-performance critical data
Table 1: Volume Types Supported by the VERITAS Volume Manager
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
9
RAID array control software exploits specialized hardware for best RAID array per-
formance. Host-based volume managers tend to offer easier manageability because
they run in richer software environments that are more closely integrated with operat-
ing systems, file systems, and applications. For example, VERITAS Volume Manager
manageability features include:
➨ Three (or more) way mirroring. This is useful when a frozen image of opera-tional data is required for backup or application testing. With three mirrored copies, one can be used for backup or testing while live applications continue to run with failure tolerant data.
➨ Snapshots, or point-in-time frozen images of volume data for ad hoc backup, query, testing, or reporting purposes.
➨ Online volume expansion coupled with file system expansion, enabling an ad-ministrator to handle unplanned storage needs without application disruption.
➨ Heterogeneous device support. VERITAS Volume Manager aggregates (mirror, stripe, concatenate or RAID) both physical disks of different types and the vir-tual disks presented by RAID array controllers.
Volume Manager Distance Limitations
Volume managers provide the robust, high-performance storage substrate that file
systems and databases need. They keep data available if disks, channels, or in cluster
environments, servers fail…as long as everything is in one computer room. RAID
arrays and volume managers are designed around highly reliable local channels that
communicate with all disks at about the same speed.
When writing to a mirrored volume, for example, a volume manager typically issues
simultaneous write commands to all disks. Application response and error recovery
algorithms both assume that all commands to devices complete in roughly the same
amount of time. Writing to a RAID volume requires more commands, but again, algo-
rithms assume that no one command takes markedly longer than any other to execute.
Similarly, all disks and channels are assumed to be equally reliable. When a disk
command fails, the volume manager retries it. If the retries fail, the volume manager
adopts permanent failure mode algorithms for the volume. For example, when one
disk of a mirrored volume fails, the volume manager transparently redirects all I/O
requests to the volume’s other disk(s).
The assumption of reliable channels and equidistant devices is valid for computer
rooms where interconnects are short and reliable. In terms of response time, all stor-
age devices are equidistant from all host computers. Business needs, however, have
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
10
created requirements for maintaining identical copies of data in situations where
channels are not reliable and where response time is not the same for all devices.
➨ Enterprises need to recover quickly from fire, flood, vandalism, power grid failure, software failure, and other events that can incapacitate entire data cen-ters. The need to recover from disasters has led organizations to establish widely separated recovery data centers. Recovery data centers are outside an expected disaster radius, and contain sufficient equipment to run critical appli-cations if the primary data center is incapacitated. For a disaster recovery site to be effective, it must have current data for critical applications.
➨ It is often useful to reuse operational and historical data without affecting online applications, for example to test new applications or to discover trends by mining data. Applications that reuse data typically require dedicated com-puters and storage, but they also need access to recent operational data.
➨ Distributed operations sometimes require that data be published from a central creation point to multiple distributed usage sites. Price lists, product specifica-tions, and web pages all fall into this category. These data must be identical throughout an enterprise.
These applications all require that data be copied between systems while it is in use,
often over links that are less reliable and have lower performance than I/O channels.
If conventional mirroring were to be used for these purposes:
➨ performance of the enterprise’s online applications would be throttled by the I/O time to write data to the remote location, and,
➨ online application availability would be reduced, since there would necessarily be an application outage each time the link between local and remote data failed momentarily.
Data Replication Technology
Data replication technology overcomes these limitations and meets business needs by
mirroring data over long distances and unreliable links. Unlike mirroring, replication
technology assumes that:
➨ writing data to remote devices can take significantly longer than writing to local devices, and,
➨ remote links are less reliable than local channels, and in particular, are subject to momentary recoverable outages.
Data replication can be configured to eliminate remote write time from application re-
sponse time by allowing I/O to be buffered locally for later delivery in case of mo-
mentary network overload or outage. Alternatively, a system administrator can con-
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
11
figure replication to maintain strict synchronization between local and remote sites.
The following sections describe data replication as implemented by the VERITAS
Volume Replicator, and how the VERITAS Volume Replicator can be used to en-
hance the disaster recoverability and utility of Oracle databases.
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
13
II. Data Replication
Protecting data against disaster
The Nature of Data Replication
Whether the purpose is reuse or disaster recovery, the mechanism of data replication
is the same. A primary replication manager intercepts live data updates at the data
processing site, and distributes them to one or more secondary replication managers
at remote sites, where they are stored persistently. Secondary replication managers
maintain a defined level of synchronization with the primary replication manager.
Replication is designed for links that are too slow to permit mirroring, or for which
transient overloads and failures are expected. Primary replication managers can queue
data updates for delayed delivery to secondary sites if necessary.
The VERITAS Volume Replicator
The VERITAS Volume Replicator, illustrated in Figure 1, is an adjunct to the
VERITAS Volume Manager. The VERITAS Volume Replicator keeps the contents
of up to 32 secondary volume groups synchronized with the contents of a primary
volume group that holds live data being processed by applications. The VERITAS
Volume Replicator maintains the integrity of replicated volumes in case of network or
system outages without significantly degrading application performance.
r
Secondary Server S1
File System /fs File A.DAT
Volume V1
File B.DAT
File System /oracle
File
Volume W1
File
Storage Replicator
Volume Manager
Secondary Server S2
File System /fs File A.DAT
Volume V2
File B.DAT
File System /oracle
File
Volume W2
File
Storage Replicator
Volume Manager
SRVM propagates application updates to volume groups at secondary sites
Primary Server P
Volume Group G
Volumes may be simple,
concatenated, striped or mirrored
Application Updates Application
Updates Application Write
Oracle RDBMS
File System /fs
File A.DAT
Volume V
File B.DAT
File System /oracle
File T1.ORA
Volume W
File T2.ORA
Storage Replicator
Volume Manager
Volume Group G1
Volume Group G2
Update Update
Update
Update Update
Update Update Update
Update
Figure 1: Replication with VERITAS Volume Replicator
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
14
In Figure 1, applications run on the primary server, accessing data on volumes in
Volume Group G. Applications may access data directly through a file system (e.g.,
/fs), or through a database management system that uses either files or raw volumes
(e.g., /oracle). In this example, VERITAS Volume Replicator replicates updates
from Volume Group G to volume groups G1 and G2 on secondary servers S1 and S2.
Figure 1 illustrates four key points about VERITAS Volume Replicator:
➨ Volume replication mirrors block updates to volume contents, whether the vol-umes’ blocks contain files used directly by applications, file systems containing databases, or raw storage used directly by a database management system.
➨ Volume replication has a single source (the primary site) and one or more tar-gets (secondary sites). Applications run at the primary site. Secondary repli-cated volumes cannot be accessed by applications while replication is active.
➨ Data are replicated over an arbitrary network. Replication does not require spe-cial hardware or dedicated network connections (although VERITAS recom-mends dedicated network links in most instances for predictable response). In principle, data can be replicated across transcontinental distances. Distance lim-its are a function of application performance needs and affordability.
➨ Although VERITAS Volume Replicator maintains identical contents, secon-dary volumes need not be of the same type as primary volumes. For example, a 3-way mirrored volume at the primary site might be replicated to 2-way mir-rored volumes, or even to non-failure tolerant simple secondary volumes.
Overview of Volume Replication with VERITAS Volume Replicator
VERITAS Volume Replicator replicates each update to a primary volume on all cor-
responding secondary volumes, regardless of the source of the request. VERITAS
Volume Replicator preserves update ordering to ensure that secondary volume con-
tents are in a correct state at all points during a sequence of updates.
VERITAS Volume Replicator is unaware of I/O request interdependencies. It is im-
possible for a file system or database management system at a secondary site to de-
termine whether data or metadata updates are in progress. This is why secondary rep-
licated volumes cannot be used by applications during replication. The after-the-fact
usage characteristic of volume replication makes it most suitable for two general ap-
plication categories:
➨ data reuse: Application testing and data mining both require access to opera-tional and recent historical data. Obviously, testing cannot be done with opera-tional data. Data mining typically requires dedicated computing and I/O re-sources, so it, too, needs its own copy of operational data.
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
15
➨ disaster recovery: A disaster recovery site is a secondary data center located far enough from the primary data center that it can continue operating if a disaster occurs. Primary site volume groups containing operational data can be repli-cated to a disaster recovery site. If a primary site disaster occurs, applications can be restarted at the disaster recovery site, and processing can resume using the (current) replicated data.
Both of these application classes have the characteristic that data at the secondary site
is used by applications after replication stops, rather than while replication is occur-
ring.1
VERITAS Volume Replicator and the VERITAS Foundation
Since VERITAS Volume Replicator replicates volume contents, VERITAS Founda-
tion Suite features are available for replicated volumes at both primary and secondary
sites, including:
➨ availability enhancements such as mirroring,
➨ general-purpose performance enhancements such as data striping, and,
➨ database-specific performance enhancements such as Quick I/O for Databases.
Policy-Based Management
VERITAS Volume Replicator is policy-based. System administrators set replication
policies to meet business requirements. Replication policies include:
➨ which volumes at the primary site are to be replicated,
➨ which volumes at secondary sites are to be replication targets,
➨ the degree of synchronization between primary and secondary sites, and,
➨ how temporary failures such as network outages are handled.
Once policies have been set, replication is automatic, requiring no system administra-
tor intervention until an exceptional event such as a site disaster actually occurs.
VERITAS Volume Replicator and the VERITAS Volume Manager
From a technical standpoint, VERITAS Volume Replicator is closely integrated with
the VERITAS Volume Manager. The executable images for both products are dis-
tributed together, although they are enabled by separate license keys. VERITAS Vol-
ume Replicator requires that VERITAS Volume Manager be installed and enabled.
1 With VERITAS Volume Replicator, secondary replicated volumes can be mirrored. Mirrors can be broken off at instants when
primary site applications are known to be quiescent, and mounted for application use at the secondary site. In this case, the bro-ken mirror of replicated data is used after replication stops, but replication of the operational copy of data resumes as soon as the mirror has been broken off and applications are re-enabled.
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
16
Synchronous and Asynchronous Replication
VERITAS Volume Replicator at the primary site logs all write requests to replicated
volumes in a Storage Replicator Log (SRL) before sending them to secondary sites.
The SRL is VERITAS Volume Replicator’s mechanism for riding through transitory
network outages and overloads.
Primary Server Oracle
RDBMS
File System /fs File A.DAT
Volume V
File B.DAT
Secondary Server
Application Updates Application
Updates Application Updates Application
Updates
File System /oracle
File T1.ORA
Volume W
File T2.ORA
File System /oracle
File T1.ORA
Volume W
File T2.ORA
File System /fs File A.DAT
Volume V
File B.DAT
Secondary SRL
Volume Ls
Storage Replicator
Volume Manager
Storage Replicator
Volume Manager
Primary SRL
Volume Lp
Figure 2: VERITAS Volume Replicator Data Flow
If a network link fails, VERITAS Volume Replicator at the primary site continues to
log updates to replicated volumes. After the network recovers, logged updates are
sent to affected secondary sites. Secondary sites with functioning network links are
updated as write requests occur at the primary site.
The behavior of VERITAS Volume Replicator during a network outage is determined
by whether the system administrator has set the volume group to replicate synchro-
nously or asynchronously.
Synchronous Replication
Replicated data at a secondary site is current if volume group contents are identical to
their primary site counterparts. If data at a secondary site is to always be current, rep-
lication must be synchronous. Each update must be logged persistently at the primary
site and received by all secondary sites before it is considered complete.
When replicating synchronously, VERITAS Volume Replicator maintains primary
and secondary site data synchronization. An application write request to a synchro-
nously replicated volume is considered complete as soon as the update is:
➨ logged at the primary site, and,
➨ transmitted to and acknowledged by all secondary sites.
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
17
Secondary sites confirm updates in two stages. The first confirmation acknowledges
receipt of the update. The second, indicating that the primary site need no longer keep
the update in its SRL, is sent when data is on disk at the secondary site.
Figure 3 is a time line for synchronous replication, showing actions that can occur
concurrently. An update to a synchronously replicated volume takes longer than an
update to a non-replicated volume by a round trip message time to the most distant (in
time) secondary site.
Application may proceed when data is
written to primary disk and received by secondary site
Analyze request
Write data to primary log
Time
Send data to secondary site
Signal completion to application
Event durations are not shown to scale
write to disk at primary site and transmission to secondary site may be concurrent
Write data to disk at secondary site
These can be concurrent
Confirm write to primary site
Application need not wait for write
to disk at secondary sites
Write data to primary site disk
Confirm receipt to primary site
Figure 3: Synchronous Replication Time Line
With the replication algorithm illustrated in Figure 3, data from write requests for
which completion has been signaled are secure against:
➨ primary site disaster, because a copy exists at each secondary site, and,
➨ secondary site or communications link failures, because all updates are logged at the primary site.
Synchronous Replication and Link Failure
With synchronous replication, an application write request is not complete until the
volume updates it implies have been acknowledged by secondary sites. This means
that application writes cannot complete if network link fails. VERITAS Volume Re-
plicator allows a system administrator to set one of three link outage policies for each
replicated volume group:
➨ signal write failure to applications,
➨ abandon replication, reverting to local-only operation, or,
➨ switch to asynchronous replication (described below).
Signaling write failure to applications effectively transfers responsibility for dealing
with the outage to them. Abandoning replication entirely requires that a new base line
copy of data be created at affected secondary sites before replication can recom-
mence. Switching to asynchronous replication allows applications to continue operat-
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
18
ing, and accumulates updates locally for later transmission to secondary sites. In prac-
tice, most VERITAS Volume Replicator installations set up to switch to asynchro-
nous replication when a link or secondary site failure occurs.
Asynchronous Replication
Application I/O peaks, network overloads, or simply a large number of secondary
sites can elongate I/O response from synchronously replicated volume undesirably.
For these situations VERITAS Volume Replicator supports asynchronous replication.
When replication is asynchronous, an application write completes as soon as
VERITAS Volume Replicator has logged the udpate in the volume group’s SRL.
Transmission and writing to secondary volumes is concurrent with continued applica-
tion execution. Figure 4 illustrates the timing differences between synchronous and
asynchronous replication.
Time
Asynchronous replication reduces application response time by this amount
Synchronous replication
Asynchronous replication
Write data to primary site log
Send data to secondary site
Confirm receipt to primary site
Signal completion to application
Application may proceed when data has
been written to primary disk and
received by secondary it
Event durations are not shown to scale
Write data to disk at secondary site
These can be concurrent
Confirm write to primary site
Application need not wait for write
to disk at secondary sites
Write data to primary site disk
Send data to secondary site
Primary site signals completion
to application
Write data to primary site disk
Write data to primary site log
Analyze request
Write data to log at secondary site
Confirm receipt to primary site
Write data to disk at secondary site
Confirm write to primary site
Application may proceed when data has been written
to log at primary site
None of these affect application response
Analyze request
Figure 4: Application Performance with Synchronous and Asynchronous Replication
As Figure 4 illustrates, asynchronous replication reduces application write response
time. The more important consequence of asynchronous replication, however, is that
it enables applications to continue to execute without performance degradation during
momentary network outages and overloads.
Asynchronous Replication and Network Loading
VERITAS Volume Replicator sends asynchronous updates to secondary sites as
quickly as network and secondary site loading permit. If a network overload is tran-
sient, updates queued at the primary SRL are eventually sent, and secondary site data
becomes current. If the network overload is chronic, the number of unsent writes
grows, and eventually, the SRL fills. When this occurs, VERITAS Volume Replicator
implements one of several policies described below. Asynchronous replication pro-
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
19
tects against short-term network overload, but it is not a substitute for inadequate
steady state network bandwidth.
Limiting Exposure with Asynchronous Replication
The advantages of asynchronous replication are:
➨ better application response compared to synchronous replication (as Figure 4 il-lustrates),
➨ application ride-through of momentary network outages and overloads, and,
➨ recoverability from secondary site crashes.
The disadvantage is that there can be (usually) brief periods of time during which data
on secondary volumes are not completely current. If a secondary computer crashes or
a communication link fails with updates logged at the primary site but not yet trans-
mitted to one or more secondary sites, secondary site data may not be current. Up-
dates in the primary site SRL are transmitted to the secondary site and written after
recovery. If the primary site suffers an unrecoverable disaster, and its SRL is de-
stroyed, secondary site recovery may start with consistent but not-quite-current data.
To limit this exposure, a system administrator can limit the number of updates by
which secondary sites are permitted to be out of date. When the limit is exceeded, ap-
plication write requests stall (no completion signal is given) until the number of un-
sent updates has fallen below a permissible threshold.
Although it does not guarantee absolute data currency, asynchronous replication is of-
ten required for performance reasons. With synchronous replication, momentary write
overload or network load from other sources can increase response times unaccepta-
bly. Asynchronous replication removes network dependencies from the application
response path, and may make replication practical where it might otherwise not be.
Storage Replication Log Overflow
VERITAS Volume Replicator’s Storage Replication Log is stored on a dedicated
volume (which should be mirrored for failure tolerance). For maximum efficiency,
the SRL is structured as a circular buffer that cannot be expanded during replication.
If a lengthy network outage or long period of heavy update activity causes the SRL to
fill, VERITAS Volume Replicator switches to keeping track locally of which regions
of replicated volumes are modified. When bandwidth is again available, VERITAS
Volume Replicator sends the contents of modified regions to affected secondary sites.
When primary and secondary sites are again synchronized, replication resumes.
Other VERITAS Volume Replicator policies for dealing with SRL overflow include:
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
20
➨ abandoning replication: Since it requires full resynchronization before replica-tion can recommence, this policy is seldom used in practice.
➨ delaying I/O request completion: VERITAS Volume Replicator will delay completion of application I/O requests until logged updates have been transmit-ted and a predetermined amount of SRL space is freed. An override causes rep-lication to be abandoned if the SRL fills while a network link is down. This cir-cumvents application failures that might result from repeated replication-related I/O request timeouts.
Getting Replication Started
Replication requires a starting point at which the contents of primary and secondary
volumes are known to be identical. Thus far, the discussion has assumed that there is
a starting point at which the contents of corresponding primary and secondary volume
groups are identical, without discussing how they might have become identical.
A replication starting point can be established by making an image (block-by-block)
backup of primary volumes (e.g., using the UNIX dd utility for small volumes, or an
image backup utility such as VERITAS NetBackup’s for larger ones). When the im-
age has been restored at each secondary site, replication can begin. This procedure
may not be workable if it is impractical to freeze primary site data until all secondary
site restores are complete.
To minimize the impact of backup-based startup, VERITAS Volume Replicator in-
cludes a replication checkpoint facility that enables volumes to be used while prepar-
ing for replication. Immediately prior to starting a primary volume image backup, an
administrator declares a replication checkpoint, causing a marker to be placed in the
SRL. When the backup is complete, the administrator places a checkend marker in the
SRL. Any updates to primary volumes between the checkpoint and checkend markers
are reflected in the SRL, bracketed by the checkstart and checkend markers. The pri-
mary volumes may be used by applications during the backup.
The completed image backup is then restored at secondary sites, and logical links
with the primary site are established. VERITAS Volume Replicator sends all updates
starting at the checkpoint marker to the secondary sites. When the checkend marker is
reached, primary and secondary volume contents are identical, and synchronous or
asynchronous replication, including update ordering, begins.
VERITAS Volume Replicator auto-synchronization can simplify replication startup
even further in some cases. To auto-synchronize, a system administrator establishes a
replication link between primary and secondary volume groups, and VERITAS Vol-
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
21
ume Replicator copies the entire primary volume group contents to the secondary
volumes. To expedite the copy, write ordering is not preserved during auto-
synchronization.
Applications may update volumes during auto-synchronization. VERITAS Volume
Replicator tracks primary volume updates by block region. When the copy is com-
plete, updated regions are re-copied. When none of their block regions differ, primary
and secondary volume groups are synchronized, and normal replication begins.
VERITAS Volume Replicator In-Band Control
VERITAS Volume Replicator maintains application write order between primary and
secondary sites to ensure that newer application updates are not overwritten by older
updates of the same blocks that arrive out of order because of network delays. This is
necessary, but not sufficient to guarantee the consistency of volume contents from a
file system or database management system viewpoint. When a file system or data-
base management system has:
➨ no transactions outstanding, and,
➨ no updated data in cache that has not yet been written to disk,
its on-disk image is said to be internally consistent. Internally consistent file system
and database images greatly simplify backup and disaster recovery.
Because VERITAS Volume Replicator deals with volumes, it cannot determine when
a database or file system’s on-disk image is internally consistent. Special in-band
control application programming interfaces (APIs), however, allow registered appli-
cations at a primary site to send messages to registered applications at secondary sites
within a replicated data stream. An in-band control message suspends updates at the
receiving secondary site until a registered application has processed it and used a
companion API to resume replication. Figure 5 illustrates in-band replication control.
Process in band control message
Invoke “resume updates” API
Write data-2 to replication log
Write data-1 to replication log
Receive in band control message
Time
Send data-1 to secondary site
Write data-1 to secondary disk
Primary site
Secondary site
Write data-1 to primary disk
Send in band control message
Write data-2 to primary disk
Write data-3 to replication log
Write data-3 to primary disk
Send data-2 to secondary site
Write data-2 to secondary disk
etc.
Write data-4 to replication log
Write data-4 to primary disk
Send data-3 to secondary site
Updates are suspended when in band control message is received
Secondary site application uses API
resume updating
Primary site continues to update replicated data
etc.
Figure 5: In-Band Control Message Time Line
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
22
A primary site application can send an in-band control message when some signifi-
cant event occurs (e.g., close of a business day). The receiving secondary site can re-
spond to the event (e.g., by creating a VERITAS Volume Manager mirror frozen im-
age of the day’s data), and then allow secondary site updates to resume.
Reciprocal Disaster Recovery
The primary and secondary site roles are specific to each replicated volume group. A
server may be the primary site for one volume group and a secondary site for another.
Figure 6 illustrates the use of this feature for reciprocal disaster recoverability.
Volume
File A2.ORA
Volume Primary Server for
Application A
Application A pdates Application A
Updates Application A Updates Application A
Updates
File A!.ORA
Application A Volumes
Primary Server for
Application B
Replication direction
Application B Replica
Volume
File B2.ORA
Volume
File B1.ORA
Application B Volumes
Application A Replica
Application B pdates Application B
Updates Application B BBUpdates Application B
Updates
Secondary Server for
Application B
Secondary Server for
Application A
Volume Group A
Volume Group B
Volume Group B’
Volume Group A’
Figure 6: Reciprocal Disaster Recovery
The system illustrated in Figure 6 has servers dedicated to applications A and B. Data
for application A are stored on Volume Group A, which is replicated to Volume
Group A’ on application B’s server, and conversely. If either site is incapacitated, its
application can be restarted on the server at the surviving site with current data.
For enterprises with large numbers of servers, reciprocal disaster recoverability based
on replication technology can dramatically improve overall IT resiliency. For a hard-
ware investment of:
➨ the storage devices required for replicated data,
➨ incremental server processing power and memory capacity to handle replication and provide adequate performance in the event of fail over, and,
➨ sufficient network bandwidth to accommodate replication traffic in addition to operational traffic,
an enterprise can position itself to rapidly resume operations when a disaster incapaci-
tates any of its critical applications.
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
23
III. Volume Replication And Oracle
Disaster protection for Oracle databases
Oracle Database Replication Techniques
By one analyst’s estimate, relational database management systems manage up to
90% of the online business information stored on servers, with Oracle holding the
largest market share. As enterprises become increasingly conscious of the importance
of their operational data and begin to plan for disaster recoverability, they naturally
consider replication of their Oracle databases.
Aware of this need, the Oracle Corporation has begun to offer a comprehensive test-
ing-based certification program by which vendors of other replication products can
verify that their products properly recover Oracle databases from site disasters, either
by themselves or in conjunction with Oracle’s own Replication Manager. This section
describes techniques for using VERITAS Volume Replicator to replicate Oracle data-
bases both with and without assistance from the Oracle Replication Manager.
Volume Replication of Oracle Databases
VERITAS Volume Replicator can replicate a complete Oracle database. Any of the
techniques described earlier can be used to establish a replication starting point with
identical database volume images at primary and secondary sites. Auto-
synchronization (page 21) uses the network to establish the starting point. For very
large databases, a full database backup coupled with the replication checkpoint tech-
nique described on page 21 may be more practical. In either case, the database may
be used during initial synchronization, but VERITAS Volume Replicator does not or-
der writes until initial synchronization is complete.
For disaster recovery, replicated database volumes must contain database control files
and redo logs as well as data files. Synchronous replication is clearly preferable to
preserve committed transactions, but performance considerations may require asyn-
chronous replication for long distances or low performance network links.
From a database standpoint, an unrecoverable disaster at the primary site is approxi-
mately equivalent to a local system crash at the secondary site. To activate the secon-
dary site database for application use:
➨ replication is stopped,
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
24
➨ secondary site replicated volumes are changed from secondary mode to primary mode and mounted, and,
➨ Oracle’s crash recovery procedures are used to restore the database image to the most current transactionally consistent point possible by applying all complete transactions from the database redo log and undoing any incomplete ones.
Oracle Standby Database Replication2
Another option for making Oracle databases recoverable is Oracle’s built-in replica-
tion facility based on database transactions rather than volume block updates. As
Oracle processes transactions, it captures them in an online redo log that can be used
to rebuild a current database from a backup if necessary due to hardware, software, or
operations failure. Periodically, the online redo log is emptied into an archive redo
log, and a new online redo log is started.
Oracle’s built-in replication facility uses these archive redo logs. Standby database
replication starts with the creation of a complete point-in-time copy of an inactive da-
tabase at a remote site. When the remote database, called a standby database, is iden-
tical to the primary database, replication begins. Each time an online redo log is
closed and copied to an archive redo log, Oracle’s Replication Manager transmits the
archive redo log to the remote site and applies the transactions in it to the standby da-
tabase. If a disaster incapacitates the primary site, the standby database is or can be
made as current as the time of the newest archive redo log transaction.
The advantage of standby database replication is that the database is always recovered
to transactional consistency. In other words, the recovered database may be out of
date (by the age of the newest archive redo log), but it does not contain partial trans-
actions (e.g., debits without matching credits), or metadata updates, and can therefore
be started and used immediately, without crash recovery.
Hybrid Replication of Oracle Databases
The disadvantage of standby database replication is that it does not restore a database
to the most current state reflected in the online redo log. Standby database replication
can be combined with volume replication of redo logs to improve the currency of a
recovered database.
2 Oracle database replication can also be used to distribute a database, allowing read/write access to data at multiple sites from
multiple sites, with user-defined policies for resolving conflicts. Since this usage is not comparable to volume replication, it is not discussed in this paper.
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
25
The most recent transactions against an Oracle database are recorded in the database’s
online redo log. The file containing this log must always be open, because the data-
base management system is continually adding to it. For performance reasons, the
must also be located near the database (i.e., at the primary site). More up to date
standby database recovery would be possible if the Oracle online redo log could
somehow be instantly transported from the disaster site to the recovery site.
Cooperation between Oracle Replication and VERITAS Volume Replicator
But transporting data instantly and transparently from a primary site to disaster recov-
ery sites is exactly what VERITAS Volume Replicator does. Allocating the Oracle
online redo log in a replicated volume makes the log available at the disaster recovery
site. In a disaster recovery scenario, after all archive redo logs have been applied to a
standby database, the replicated online redo log can be applied to roll the database
forward to the point of the last complete primary site transaction.
The Oracle Storage Compatibility Program
Up to date database recovery after a site disaster is obviously extraordinarily valuable
to enterprises. Recognizing this, Oracle Corporation has instituted an Oracle Storage
Compatibility Program (OSCP), an audited program of pre-defined tests that verify
the ability of a data replication solution to support up to date transactionally consis-
tent recovery of Oracle databases at remote sites.
Any storage configuration that supports replication of Oracle databases is eligible to
participate in OSCP, including both “hardware” solutions that replicate data between
disk controllers, and hybrid solutions such as VERITAS Volume Replicator, that
combine basic storage hardware with host-based software. VERITAS has submitted
the hybrid VERITAS Volume Replicator/Standby Database Replication technique de-
scribed above to the OSCP testing process, and is the first vendor to successfully pass
Oracle’s tests. VERITAS Volume Replicator is approved by Oracle Corporation for
up to date disaster recovery of Oracle databases.
Comparison of Oracle Replication Techniques
Whether standby database replication augmented by the volume replication of the
Oracle online redo log, or full replication of the volumes holding the database and
other application data is preferable depends on enterprise priorities. For example, if a
database must be replicated across thousands of miles, data propagation time might
preclude synchronous volume replication because of the adverse effect on application
responsiveness. Similarly, if database corruption by unstable applications is a con-
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
26
cern, the standby database technique with delayed application of archive redo logs
would probably be the preferred solution. On the other hand, if rapid recovery from a
primary site disaster is the main concern, volume replication is probably the best solu-
tion, because the recovery process is less time consuming and complex than with
standby database recovery. Table 2 compares seven key aspects of the hybrid and full
volume replication techniques.
Characteristic Standby Database Replication with Volume Replication for
Online Redo Log
Volume Replication of Entire Database
Cost Potentially lower communication cost because archive redo logs are batched (greater efficiency) and are not latency-critical
Potentially higher communication cost because synchronous replication I/O is in the application response path and therefore latency is critical.
Replication Granularity Individual tables Contents of a volume (usually more than one table)
Protection againstDatabase Corruption
Better, because application, operating system, or storage subsystem-caused corruption is not reflected in the standby database until archive redo logs are applied.
Any application, operating system, or database management system-caused corruption is reflected immediately in the secondary image.
Database OperationalPerformance
No impact on database I/O; minor impact possible when archive redo logs are copied and sent to secondary site; some impact if replication of the online redo log is synchronous.
Each write request incurs SRL write delay plus message round trip time (for synchronous replication only)
Failover Time(time until applications
can begin using databaseat secondary site)
Possibly longer. All archive redo logs not yet applied to the standby database must be applied, and the online redo log must be replayed.
Typically shorter. Treated as an Oracle crash recovery. Only the online redo log need be replayed.
Failback Time(move operational
database back to originalprimary site)
Typically much shorter if database image at primary site survived the disaster. Archive and redo logs recorded since failover must be sent to the primary site and replayed against the database image there.
Typically longer. Image of operational database at secondary site must be created at primary site. If operational database is used by applications during restore, a switch of primary sites is also required.
Management Simplicity Potentially more complex to configure; both VERITAS Volume Replicator and Standby Database Replication must be configured (Oracle8i simplifies setup).
Probably simpler. Worst case disaster recovery scenario is approximately equivalent to Oracle local crash recovery.
Table 2: Comparison of Standby Database Hybrid and Pure Volume Replication3
Another Hybrid Replication Alternative
A variation of the hybrid alternative described above is to store both online redo log
and archive redo logs on volumes replicated at the recovery site. With this solution,
3 Table 2 is based on the paper Guidelines for Using Remote Mirroring Storage Systems for Oracle Database by J. Bill Lee, pub-
lished by Oracle Corporation.
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
27
archive redo logs and the online redo log are both immediately available at the secon-
dary site, so there is no possibility of unsent archive logs being destroyed in a primary
site disaster. Archive logs can be applied at the secondary site by creating and mount-
ing VERITAS Volume Manager split mirrors of the volumes containing them.
This solution also allows the replaying of archive logs against the standby database to
be deferred. This provides the same protection against application, operating system,
or database management system-caused database corruption as conventional standby
database replication with the Oracle Replication Manager managing the transmission
of the archive replication logs over the network.
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
29
IV. Summing Up
Volume replication in context
VERITAS Total Data Storage Management
VERITAS Volume Replicator is just one component of VERITAS Total Data Stor-
age Management. Total Data Storage Management consists of:
➨ a solid foundation including a flexible, failure tolerant volume manager and an enterprise-class file system,
➨ automated backup and media management for distributed enterprise require-ments,
➨ clustering for automated continuous application availability.
Why the VERITAS Solution?
Five factors qualify VERITAS uniquely to provide total data storage management
solutions for the enterprise:
➨ technology leadership,
➨ product integration,
➨ multi-platform support,
➨ support, training, and service, and,
➨ experience and reputation.
Technology Leadership
Since the company’s founding in 1992, VERITAS has been an innovator in enterprise
data storage management software technology, with foundation technologies such as:
➨ high-performance, failure tolerant, logical volumes,
➨ extent-based file allocation,
➨ online volume and file system expansion and defragmentation,
➨ several frozen image technologies for volumes and file systems,
➨ direct I/O for database management systems (Quick I/O for Databases).
In backup, VERITAS Block Level Incremental Backup (BLIB) has enabled fast, non-
intrusive backups of Oracle databases. VERITAS clustering is also making highly
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
30
available enterprise computing a reality. Volume replication is only the latest tech-
nology in which VERITAS is innovating, to provide practical disaster tolerance.
As the data storage management technology leader, VERITAS is actively driving new
I/O architecture standards. Working through organizations like the Storage Network-
ing Industry Association, VERITAS is helping to create open technologies for SAN
management, for file system enhancements for effective SAN use, for increased
backup efficiency, for intelligent storage devices, and in a variety of other areas. Be-
cause VERITAS is co-developing these technologies, the company will be among the
first to bring their benefits to users in future data storage management products.
Product Integration
Product integration is the second factor that makes VERITAS the right choice for to-
tal data storage management. Examples of VERITAS product integration include:
➨ File system and volume manager cooperation that enables online expansion and defragmentation, snapshots, and Quick I/O for Databases.
➨ In addition to the database replication capabilities described in this paper, VERITAS Volume Replicator cooperates with the VERITAS Cluster Server and Global Cluster Manager to make cross-cluster disaster recoverability and global applications a practical reality.
➨ VERITAS Cluster Server agents make VERITAS File System, VERITAS Vol-ume Manager, VERITAS Volume Replicator, and VERITAS NetBackup inte-gral parts of a failure-tolerant data storage management environment.
VERITAS develops new products and enhances existing ones with an awareness that
integrated data storage management is much more compelling to users than a collec-
tion of unrelated or loosely related products. This thinking causes product decisions
to be made in the context of integrated solutions.
Multi-Platform Support
The third reason to choose VERITAS total data storage management is the com-
pany’s support of major UNIX and Windows server operating systems. With a
VERITAS Total Data Storage Management strategy, the most important IT asset,
skills and knowledge, is applicable across the entire enterprise. Adding or changing
computing platforms does not require new data storage management skills.
Support, Training and Service
The fourth reason to choose VERITAS is the company’s extensive network of cus-
tomer support, service, and training facilities. VERITAS products are applied to some
of today’s most difficult IT problems, where the answers aren’t always obvious.
VERITAS Volume Replication and Oracle Databases Edition: 5/29/00 8:50 PM Copyright VERITAS Software Corporation, 1999,2000 by: Paul Massiglia
31
VERITAS training gives users the knowledge to design optimal data storage man-
agement architectures. VERITAS consulting services can help plan, design and im-
plement turn-key information services. And VERITAS service is a phone call away if
something does go wrong.
Experience and Reputation
The final, and perhaps most compelling reason that makes VERITAS the right data
storage management choice is the company’s experience and reputation in large en-
terprises. Enterprise computing is a tough business. Only the best survive. VERITAS
has not only survived, but has become the acknowledged leader in enterprise UNIX
and Windows data management, with over 60% of the Fortune 1000 as its customers.
VERITAS’ reputation for solid, dependable, yet innovative data storage management
solutions gives customers the confidence to adopt the company’s solutions in new
technology areas such as volume replication to provide heretofore unavailable IT
quality of service. Being first to deploy these advanced capabilities gives VERITAS
customers a competitive edge in their fields.
A VERITAS data storage management solution isn’t just a set of products, it’s a
company with the technology, vision, products, experience, and resources to make IT
the robust, responsive service it needs to be for success in today’s competitive world.
Business without Interruption