zero data loss recovery appliance best practices

Zero Data Loss Recovery Appliance Deployment Best Practices

Donna Cooksey Oracle Principal Product Manager

Jony Safi Oracle Principal Member of Technical Staff

September 30, 2014

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |


Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Program Agenda

Overview

Architecture, Sizing and Performance

Getting Started

Managing 100s Database Backups

Summary

1

2

3

4

5


Zero Data Loss Recovery Appliance (Recovery Appliance) Product Overview

5


Zero Data Loss Recovery Appliance Overview

6

Recovery Appliance Delta Store • Stores validated, compressed DB changes on disk • Fast restores to any point-in-time using deltas • Built on Exadata scaling and resilience • Enterprise Manager end-to-end control

Recovery Appliance

RMAN Delta Push • DBs access and send only changes

• Minimal impact on production • Data Guard-like real-time redo ship

instantly protects new transactions

Protected Databases

Protects all DBs in Data Center • Petabytes of data, any DB release • No expensive DB backup agents

Offloads Tape Backup

Replicates to Remote Recovery Appliance


Real-Time Recovery Status for All Protected Databases Database Level Recoverability

Partial screenshot or Recovery Appliance Protected Database Page:

Complete Database Recovery Spectrum at your finger-tips!

Current Point-in-time-recovery (PITR) capability for the database from Recovery Appliance backups Current Unprotected Data Window for the database

7


Architecture, Sizing and Performance Recovery Appliance


Pre-built and Optimized Out-of-the-Box Av

aila

bilit

y Ac

hiev

emen

t

Time (Days)

100%

Avai

labi

lity

Achi

evem

ent

Time (Months)

Test, Diagnose, tune and

reconfigure

Test & debug failure modes

Assemble multiple

components

Multi-vendor finger

pointing

With I.T. Custom Configuration With Recovery Appliance

9


ZDLRA - X4 Hardware Configuration • Base Rack:

– 2 Compute Servers with high speed connectivity • 4 * 10 Gb Ethernet ports per server • Dual 40Gb/s InfiniBand ports per server • Dual-port 16Gb Fibre Channel for tape connectivity (optional) per server

– 3 Storage Servers • Each storage server has 12 high capacity disks

• Increase capacity by adding storage servers • Full Rack: 2 Compute and 14 Storage Servers

Fully Redundant

Extra Storage

Recovery Appliance Base Rack

10


Standardized Configuration Single-Vendor Technical Support

• Recovery Appliance software stack – Embedded and optimized in Oracle 12c Database – Oracle database houses Recovery Appliance

metadata and RMAN catalog – Oracle Secure Backup bundled for tape media

management – Exadata software optimized for Recovery

Manager

• Scalable Exadata platform – 80Gbits InfiniBand internal network bandwidth

(Per Compute server) – 16Gb Fibre Channel cards for tape connectivity

(optional) (Per Compute server) – Exadata Smart Flash Optimized for Backups

• Exadata HA Platform EM Cloud Control 12c

11

Recovery Appliance / RMAN plug-in on client databases

Delta Store

Catalog


Storage Configuration

• All protected databases leverage the Recovery Appliance catalog

• Two pre-defined disk groups: – - High redundancy – - Normal redundancy

• On Storage Location (Delta) is pre-configured and leverages all usable space on the appliance – The storage location may be easily expanded as

additional capacity is needed

Storage Location(s) • Logical storage container:

• Database backups • Archived log backups

• Space dynamically shared based on user-defined recovery settings

• Shared by all / many databases

Catalog Delta

Recovery Appliance Catalog

Fast Recovery Area • Catalog backup and archived logs • Incoming redo from protected

databases

12


Compression and Backups on Recovery Appliance

• Compressed production data remains compressed in the backup – This data will not benefit from further compression during the backup

(e.g. RMAN backup or Recovery Appliance compression) – Deduplication software cannot deduplicate compressed data

HCC Data OLTP Compressed Tables

SecureFiles Compressed/Deduplicated

• RMAN compressed backups are not needed in Recovery Appliance environments – Recovery Appliance compresses backups – Incremental forever backup strategy reduces network traffic

13


Recovery Appliance Backup Module

• The SBT Library is bundled with the Recovery Appliance database • Protected (client) databases, can either:

– Download the Recovery Appliance Backup Module from OTN – Install the module from the Recovery Appliance ORACLE_HOME/lib directory

• Install the Recovery Appliance backup module on the protected database host file system of the protected database – $ORACLE_HOME/lib is default location for shared libraries for the Oracle database – When the backup module is installed, an Oracle wallet should be created to store the

credentials needed to authenticate the protected database with the Recovery Appliance

14


Connectivity • Recommend 10GbE for backup / restore transport

– IB connectivity is supported – All performance benchmarks use the recommended 10GbE network

• Network encryption – Can use SSL (TCPS encrypted while in transit ) – Currently is a manual step

• How To Configure Scan Listeners With A TCPS Port (Doc ID 1092753.1) • Step by Step Guide: How to Configure SSL / TCPS on ORACLE RAC (with SCAN)(Doc ID 1448841.1)

15


Recovery Appliance Sizing - Methodologies

Comparative Sizing (vs. an Existing System) Predictive Sizing (for a new deployment)

Sizing estimate based on current utilization: - Capacity and throughput of existing backup

appliance(s)

Holistic approach to centralizing and standardizing Oracle database backups

Considers capacity and throughput differences based on standardization of backups leveraging an incremental forever strategy

Starts with a “clean slate” to architect a new centralized backup management strategy

16


Recovery Appliance Sizing – 5 Steps Initial and Ongoing Capacity Planning

Define Define the Recovery Appliance size based on the collected information.

Collect Collect existing database’s information

Validate Validate the collected data

Discuss Discuss the results; tune & finalize

Monitor Use the Recovery Appliance Capacity Planning Reports

17


Recovery Appliance Sizing

• Sizing estimates is a matter of doing the math: – Size and growth rate of database

• Minus temp, free space, committed undo

– Percent of database that changes each day – Redo generation rate – Desired recovery window in days – Compressibility of database – Annual data growth rate – System lifecycle (e.g. 3 years)

18

224 TB Usable Delta Space/ Rack 2.2 Petabytes of Virtual Backup Space


Performance and Scalability

• Add InfiniBand connected racks to scale – Add spine switch when connecting additional

rack

• Can expand with new generation hardware

• Scale to 18 Racks – Up to 216 TB/hour Delta Ingest and Restore

19

Recovery Appliance

Up to 120 TB/hour Virtual Backup Rate Up to 12 TB/hour Sustained Delta Ingest

Up to 12 TB/hour Restore Rate

Performance Per Full Rack


Performance Considerations • The Recovery Appliance is tuned out-of-the-box for optimal performance • Monitor replication rates on the EM home page and BI reports

– Increase or decrease replication channels (streams) as appropriate

• Attach tape drives to the appliance up to throughput saturation of Fibre cards – Up to 12 StorageTek T10000D and 20 LTO6 tape drives per rack

• Total network throughput per rack is 80GbE / sec – Separate interface assignments by activity (e.g. separate replication traffic from ingest

traffic if possible)

20


Getting Started Recovery Appliance


Recovery Appliance User Accounts

• The Recovery Appliance administrator creates database user accounts on the appliance – These are called VPC users. – Privileges to send and receive backups for one or more protected databases

• Each Recovery Appliance user owns a virtual private catalog (VPC) for accessing metadata in the catalog for only those databases

• The protected database’s administrator with SYSDBA or SYSBACKUP privileges is associated with the Recovery Appliance user

• Authentication credentials of the Recovery Appliance user are stored securely in an Oracle Wallet on the protected database host – Database administrator connects to Recovery Appliance VPC user using the catalog role

DBAs May Only Access Metadata For Their Databases

22


Recovery Appliance Catalog

• Import existing backup metadata into the Recovery Appliance catalog – Upgrade RMAN recovery catalog to Oracle Database 12.1.0.2 – Use the RMAN IMPORT CATALOG command to import backup metadata

• Ensure backups / restores are not in progress during catalog import

• Dual catalog approach – Keep existing RMAN catalog and use Recovery Appliance catalog for backup metadata under Appliance management

• Automated catalog backups – Daily to Recovery Appliance CATALOG disk group

• If tape drives are attached, backups to tape are automated as well

Used by All Protected Databases Under Management

23


Backup Migration Strategy to the Recovery Appliance

• Migrate existing protected databases to the Recovery Appliance – Migrate the most recent RMAN level 0 backup and subsequent level 1s as well as

archived log backups • Migrate via Recovery Appliance polling capability or using RMAN BACKUP AS BACKUPSET

• Cut-over strategy: – Begin backup operations to Recovery Appliance on selected date

• Restores requiring backup pieces prior to the cut off date, will use previous backup location if the backup pieces have not been fully migrated to ZDLRA yet.

Methodologies

24


Enterprise Manager Prerequisites • Enterprise Manager is the recommended interface for the Recovery Appliance and

includes the following prerequisites: – Enterprise Manager Framework 12.1.0.4 – Recovery Appliance plug-in 12.1.0.1 – Database plug-in 12.1.0.6 – Exadata plug-in 12.1.0.6

• Feature availability by Oracle Database version: – Backup to the Recovery Appliance configured in EM Backup Setting Page for Oracle Database 11.2 forward – Real-time redo transport may be configured in this page for Oracle Database 11.2.0.4 forward – EM automated protected database wallet configuration Oracle Database 11.2 forward – Oracle Wallet is manually configured for Oracle database 11.1 and 10.2 prior to enrolling the database with the

Recovery Server

25


Managing 100’s of Database Backups Multi-Tiered Recovery Appliance Environments


Unified End-to-End Control

• Appliance Admin monitors and manages all centralized backup activities

• Database Admins monitor the protection status of their databases from disk, to tape, to replica – Offloaded replicas and tape backups

appear in Recovery Catalog

27

Tape

Remote Appliance

Enterprise Manager

Tape


Database Protection as A Service • Databases are managed as a

group (by policy) to define: – Replication – Scheduling copy of backup to tape

or cloud

• Same or different policy on remote Oracle Recovery Appliance

• Reporting by policy and database

• EM Group(s) may be used to quickly add all databases within the group to the Recovery Appliance

Protection Policy

- Defined Class of Service

• Recovery requirements − Disk − Tape or Cloud

• Maximum on disk retention

• Prioritization of backups • Polling local backups • Unprotected data

window threshold

Business SLAs categorized: - One policy per database tier

28

Databases


Protection Policies – Defines a Class of Service

• One protection policy associated with many databases – Define one protection policy for each Class of Service:

• Disk recovery window goal and Backup Copy Policy • Optionally - tape recovery window, unprotected data window threshold, maximum disk retention or

backup polling settings

• If unprotected data window threshold defined: – Ensure all databases associated with the policy have real-time shipping enabled or

backup on the same frequency

• Replication is configured by protection policy so all databases associated with a protection policy should have the same replication requirement

Foundation of Database Protection as A Service

29


Prioritization of Incoming Backups or Recovery Goal

• Extreme space pressure should NEVER occur! – Could only happen if the Recovery Appliance administrator IGNORES the numerous alerts and warning

• Backup Copy Policy setting in the Protection Policy defines prior IF extreme space pressure occurs: – Always accept new, incoming backups (default)

• Oldest backup within the recovery window could be purged to make room for incoming backups

– Refuse new, incoming backups if not enough space to meet recovery window • Backups within recovery window could only be purged on disk, if they’d been copied to tape or replicated • If this policy is enabled, backups on the Recovery Appliance will never have more than their reserved space setting

Only Applicable if Extreme Space Pressure Occurs

30


Standardization Across the Enterprise Autonomous Management of All Backup Assets

Weekly full

Bronze Policy (Test / Dev)

Silver Policy (Business-Critical)

Gold Policy (Mission-Critical)

Replicate

Weekly full / daily incremental

Weekly full

Policy

Policy

• Recovery Window = 7 days on disk, 90 days on cloud or tape • Maximum disk retention = 8 days • Copy of backup required on alternate media • Unprotected Data Window Threshold = 2 minutes (Example)

Protection Policy

31


Automated Backup Polling

• Recovery Appliance can copy or migrate local RMAN backupset backups – Backup polling directory and frequency (schedule) is defined in a

Protection Policy and applicable to all databases associated with the policy

• Backup polling can be effectively used to migrate existing RMAN backupset backups into the Appliance – Extends point-to-time recovery capability from the Appliance to

that of the oldest backup copied • Avoids having to perform the initial full backup to seed the Appliance

• Mount the Network File System (NFS) directory that stores backups

Copy or Migrate Local Backups to the Recovery Appliance

Local backups in shared directory

Polling Location

32


Adding a Protected Database To Recovery Appliance

Schedule backups using the Oracle Suggested Strategy for Recovery Appliance

Two Steps Plus a Backup Schedule

EM - ZDLRA Protected Database Page

Click the “ADD” Button to evoke the EM wizard to: • Add one or more databases • Associate the database(s) with a Protection Policy • Define estimated storage need (Reserved Space) • Select or define the appropriate user credentials

EM - Database Backup Settings Page • Select the appropriate Recovery Appliance

from the dropdown menu • Select the appropriate VPC user • Click check box to enable real-time redo

shipping • Define RMAN parallelism via “Tape Settings”

as an SBT library is used

33


Eliminate Storage Over-Provisioning Guaranteed Storage Allocation: Only if Needed to Meet Recovery Window Goals

34

Balances individual database and aggregate of all database recovery goals.

Disk Storage Projected to Meet Recovery Goal -Recovery Appliance " Reserved Space”

- Guaranteed storage amount provided IF needed to meet Recovery Window Goal

Business Requirement - Recovery Window Goal

Protected Database Requirements

All Recovery Window Goals

Storage is dynamically shared across all databases within the location to meet

recovery window goals.

Oracle ZDLRA Storage Location


Backing up a Protected Database • The RMAN backup destination to use depends on the configuration:

– “SBT_TAPE” destination when backing up directly to Recovery Appliance and “DISK” when using Recovery Appliance polling settings

• Backup and restore operations must connect to the Recovery Appliance catalog – Utilize RMAN CONNECT as TARGET for protected database and as CATALOG for communication with the

Recovery Appliance • Pluggable databases-connect to the root of the multitenant container database (CDB) as TARGET

• Backup retention to disk and tape is configured via Recovery appliance protection policies – Do not utilize RMAN “DELETE OBSOLETE” command

• Protected database configuration recommendations: – Enable Block Change Tracking for fast incremental backups – Utilize a local Fast Recovery Area for archived logs, online redo, Flashback logs

35


RMAN Recovery Processes Same with Recovery Appliance

• RMAN restore and recovery commands identical between traditional recovery and modern Recovery Appliance environments – RMAN backup strategy changes to use incremental forever strategy – Same level 1

backup commands – Granularity of recovery from block media to full virtual restore

• Oracle Data Recovery Advisor (DRA) – Enables database recovery by using automatic repair actions

• User Directed Recovery – Manual recovery based on specified criteria

36


End-to-End Visibility and Management

• Managing databases as a group increases consistency and manageability: – Replication is defined by Protection Policy

• Incoming backups of databases associated with the protection policy are immediately replicated – No scheduling required

• If replication activity is paused by the user, upon resuming replication all backups since the pause would be automatically replicated

– Copy to tape jobs may be scheduled by database or protection policy

• When initially configuring replication or a copy to tape job, the most recent virtual full backup will be copied (e.g. point of last incremental)

Multi-Tiered Environments: Copy to Tape and / or Replica

37


Centralized Copy to Tape

• Oracle Secure Backup (OSB) pre-configured with default settings: – Recovery Appliance compute servers are OSB

media servers for direct attached tape devices – Tape pools (OSB media families) – Automated OSB catalog backup – Automated discovery and configuration of

attached tape devices

• Easily customize for advanced tape management (e.g. tape duplication and vaulting etc)

Out-of-the-Box with Oracle Secure Backup

Tape Library

Autonomous Tape Archive • Offloads tape backup • Tapes utilized all day

38


Managing Copy to Tape Operations

• Defines the recovery window on tape

Recovery Appliance Protection Policy

Defines:

• Specific database or all databases in a protection policy

• Type of backup - Full, Incremental and / or Archived logs

• Job priority, # of copies and runtime window

• Media Management Library and Attribute Set

• Copy to tape schedule – one time or recurring

Copy to Tape Job Templates

• Associated with a Media Manager Library

• Define number of streams to use for the backup operation

• Media management parameters (optional)

– These will override any defined in the Media Manager Library setting or via the media manager

Attribute Set (s)

• Associated with installed media manager specific SBT library (OSB by default)

• Sets maximum number of RMAN channels which can be used concurrently (based on # of available tape drives)

• Define RMAN media parameters to be used

Media Manager Library

39


Replication for Disaster and/or Site Failure Protection

Topology Requirement Recovery Appliance Replication

Traditional

Require backup copies to be retained at an offsite location for disaster recovery

Bi-Directional

Multiple data center environments in which each is a disaster recovery site for the other

Hub and Spoke

• Multiple remote data centers/offices

• Backups are aggregated to one central location

Recovery Appliance Supports Most Replication Topologies

ZDLRA 1 (Source)

ZDLRA 2 (Replica)

Ingest Process

Ingest Process

Incr Backup

Incr Backup

Upstream (source) catalog is updated with metadata regarding copies on the Recovery Appliance replica

40


Replication

• Setting up replication is very easy: – Create a Recovery Appliance user for replication – Create an Oracle Wallet on both upstream and downstream appliances for

authenticated communication

• Add protected databases to the to the downstream Recovery Appliance, then upstream

• Ensure the replication user is granted access to the database on the downstream Recovery Appliance

Set-up as Easy as 1, 2, 3

41


Recovery Appliance and Data Guard • Backups to the Recovery Appliance may be performed from the primary or

standby database – Similarly copy to tape operations may be performed from the upstream or

downstream Recovery Appliance

• In Oracle Database 12c environments, Data Guard Far Sync can be used to ship redo to the Recovery Appliance and Active Data Guard standby database(s)

• Recovery Appliance replicates archived log backups versus redo

42


• Two distinct benefits of how Recovery Appliance leverages real-time redo: – Reduces data loss exposure to a sub-second level – Offloads backup of archived logs from production

database server

• Recovery Appliance is configured as an additional redo destination for protected databases – Oracle Database 11.1.0.7 forward*

*Platform and advanced capabilities may vary by database version

Redo

Archived Log Backups

Log switch occurs

Online redo

Recovery Appliance

Real-Time Redo Transport

• How use of redo differs: – Recovery Appliance backs up the archived logs – Data Guard applies the redo to a standby database

43


Advanced Monitoring and Alerting • The Recovery Appliance home page provides a dashboard of current status

– Warnings and alerts are prominently displayed which may need user attention

• Effectively manage any issues, assign ownership and track problems through to resolution – Leverages the EM incident and event notification system

• Default and user-defined metrics trigger warnings or error messages based on thresholds – EM collects metrics on key information providing out-of-the box monitoring and

alerting

44


Recovery Appliance Home Page in Enterprise Manager

• Sections include: – Summary of current and

recent activity – Protected database

backup/recovery issues – Daily data sent / received – Performance – Media Manager status – Replication status to / from – Storage Location status – Incidents and Events

Dashboard for Current Status of the Backup Environment

45


Management Reports

• Protected Database Details

• Recovery Window Summary – Recovery status by protection

policy

• Top 10 Protected Databases by Data Transfer

• Capacity Planning Reports: – Summary – Details

Generate on a Recurring Schedule or on an Ad hoc Basis Protected Database Details Report (Partial screenshot):

46


Standardized Oracle Database Protection

• Database Protection as A Service – Defined class of service whereas databases may be managed as a group for multi-

tiered protection strategy – Reports by database, appliance and protection policies

• Incremental forever backup strategy for all Oracle databases in the data center

• Shifts focus from retention to database recoverability • Pro-active capacity monitoring to meet current and future requirements

Reduces Complexity and Improves Compliance

47


Summary Recovery Appliance

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 49

ZDLRA Unique Benefits for Business and I.T.

Minimal Impact Backups Production databases only send changes. All backup and tape processing offloaded

Zero Data Loss Real-time redo shipping provides instant protection of new transactions

Cloud-Scale Protection

Easily protect all databases in the data center using massively scalable service

Database Level Recoverability

End-to-end reliability, visibility, and control of databases, not disjoint files


Key HA Sessions and Demos by Oracle Development Monday, 29 September, Moscone South

11:45 MAA with Oracle Multitenant – Seeing is Believing, 104 1:30 Oracle Database 12c HA for Consolidation and Cloud, 306 2:45 Zero Data Loss Recovery Appliance, New Era in Data Protection, 307 4:00 Oracle GoldenGate 12c for Oracle Database 12c, 305 5:15 Maximizing Oracle RAC Uptime, 103 Tuesday, 30 September, Moscone South 10:45 Active Data Guard and GoldenGate HA Best Practices, 308 12:00 Zero-Downtime Mantra for Applications with Oracle RAC, 309 3:45 Zero Data Loss Recovery Appliance Best Practices, 305 5:00 Oracle WebLogic Server 12c: Oracle Database Integration, 304 5:00 Geodistributed Oracle GoldenGate and Active Data Guard:

Global Data Services, 307

Wednesday, 1 October, Moscone South 10:15 Resource Manager Best Practices 11:30 RMAN Best Practices in Oracle Database 12c, 104 12:45 Active Data Guard: Best Practices and Deep Dive, 104 2:00 Expert High-Availability Best Practices for Oracle Exadata, 102 4:45 GoldenGate Performance and Tuning for Oracle, NORTH 130

Thursday, 2 October, Moscone South 9:30 Best Practices for Zero Downtime, 103 12:00 Data Protection,Recovery and HA for Private Cloud, 103 Demos – Moscone South

Oracle Maximum Availability Architecture, SLD-140 Oracle Active Data Guard, SLD-145 Global Data Services, SLD-144

Continuous Availability, SLD-125 RMAN, Database Backup Cloud Service, Flashback, SLD-141 Oracle Secure Backup, SLD-142 Oracle Real Application Clusters, SLD-128

oracle.com/goto/availability https://blogs.oracle.com/MAA @OracleMAA

zero data loss recovery appliance best practices

Documents