zdlra deep dive

BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA

HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

ZDLRA – Deep Dive

Daniele Massimi – Principal Consultant

BE-IMS

Agenda

ZDLRA - in Action2 21.06.2017

1. ZDLRA – Functionality under the hood

2. Installation and Configuration from bottom up

3. Have a look inside – (just particularities)

4. From protected Databases to backups

5. Backup and Recovery Capabilities

6. Recovery Appliance Views and Utilities

7. Conclusion

Who am I


Daniele Massimi, Trivadis AG (CH, Bern)Principal Consultant [email protected]

Seit 2012 bei Trivadis IMS (Infrastructure Service Management)

Oracle DBA seit 2000 (Exadata > 6 Jahre)

Infrastruktur-Architekturen, Operations & Administration, Upgrades and Migration, High Availability, Backup & Recovery, Virtualizierung

Engineered Systems (Exadata, ODA, ZDLRA, PCA)

Trainer (Oracle Architectur and Internal, 12c New Features, Exadata)

Oracle CertificationsOracle Certified Master (11g)Oracle Certified Professional (8i – 12c)Oracle Certified Expert (RAC)Oracle Implementation Specialist (Exadata, OVM)

mailto:[email protected]

Our company.

© Trivadis – The Company4 21.06.2017

Trivadis is a market leader in IT consulting, system integration, solution engineering

and the provision of IT services focusing on and

technologies

in Switzerland, Germany, Austria and Denmark. We offer our services in the following

strategic business fields:

Trivadis Services takes over the interactive operation of your IT systems.

O P E R A T I O N


ZDLRA

Functionality under the Hood

Common problems with Oracle backups


RPO depends on archive backup frequency (many minutes or hours)

Backup window and nightly batches compete for resources

High network bandwidth usage for full backups

Low-speed backups and restores

Backup and restore success ratio penalized by shared infrastructure

Critical databases require Data Guard for transaction safety

– Additional storage space, license and computation required

Expiration of backup set on ML: need to crosscheck and delete expired

Lack of backup validation

…

Recovery Appliance Architecture


Source: Oracle

Minimal Backup Overhead

8 21.06.2017

Traditional Backup

– Read/Write for Disk/Tape Backup

– Deduplication

– Compression

– Validation

– Deletion

– Merging

All on database host

Degrade performance

Recovery Appliance

– Offloading operations

– Delta Push

– Delta Store

Source: Oracle

Delta Push


Delta Push is a highly optimized form of source-

side optimization

– Through RMAN block change tracking

– No reading of unchanged data

Two operations on the protected Database

– Incremental "forever" backup

– Real time redo transport

One-time full backup as prerequirement

Afterwards Incremental forever backup

Validate incoming Backups against corrupted

physical blocks

Compress the Backups using special block-level

Algorithm

Backuped Database

Changed Data

Less data, less I/O, less network traffic

Source: Oracle

Eliminate data loss – Real Time redo transport


Real Time redo transport

– Data Guard like but asynchronous

– Changes out of Logbuffer transfered

– Validate and writes to a stage area

Redolog switch on database

– RA converts the staged redo data to a

compressed archived redolog backup

– Writes this archlog backup to catalog

– Ready for future restores

Explicit Archivelog backup no more necessary

Reduces RPO to (near) zero

Source: Oracle

Delta Store – the “brains” of the Recovery Appliance


Backup

Totality of all Database Backup in a

Storage Location

validates the incoming blocks

compresses, indexes and stores

deduplicate, less storage usage

High number of virtual full backups

Higher recover window

Delta Store

Day NIncr

Day 1 Virtual Full

Day N Virtual Full

Day 1Incr

Day 0Full

Source: Oracle

Delta Store – efficient restore/recover


Restore

Creates a Virtual Full Database

Backups

No need of transfer and apply

incremental backups during restore

operations

Roll forward with restored redologs

Uses Exadata performance and

scalability

Delta Store

Day 0Full

Day 1Incr

Day NIncr

Day ‘N’ Full Backup

Source: Oracle

Delta Pools


Chunks inside the Delta Store

Backups will be indexed and stored inside the Delta Pools

Set of datafiles blocks from which RA constucts Virtual Full Backups

Each backuped datafile will have his own delta Pool

Automated delta pool space management

– Protection Policy will be inherited to database retention policy

– Delete automatically expired or obseleted Backups

– Periodically reorganising for performance improving

– Delta pool optimization when old blocks are deleted and new incremental Backup

arriving

Source: Oracle

Delta Push – how it works (1/2)


Clients address the HTTP Servlet on ZDLRA

RMAN ships incremental Backup to Staging Area

Data will be validated and Metadata will be stored on RAC Database

Data Blocks of Backupsets will be indexed and stored on Delta Store

Source: Oracle

Delta Push – how it works (2/2)


Optionally Real Time Redo Transport could be activated

ZDLRA will be using Data Guard Technology (RFS on RA)

Validated Redo Blocks will be stored on Delta Store

Archive Logs will be generated whenever a Log Switch happen on Prod DB

Optionally you can replicate or copy to a remote ZDLRA or Tape

Source: Oracle

Restore – how it works


Clients address the HTTP Servlet on ZDLRA

ZDLRA re-assemble and validate "virtual Backupsets"

Data Blocks will be shipped back to Client

Source: Oracle

Policy-Based Database Protection as a Service

ZDLRA Workshop - Introduction17 21.06.2017

Standardized

protection policies

– Recovery window

– Retention

– Replication

Gold Policy – Customer Critical

Disk: 45 daysTape: 90 days

Silver Policy – Internal Critical


Bronze Policy - Test/Dev


Protection Policies Attributes

ZDLRA Workshop - Protection Policies18 21.06.2017

Storage Location Recovery Appliance storage location for storing backups

Recovery Window Disk Tape recovery window goal for the protected database

Recovery Window Tape Tape recovery window goal for the protected database

Max Retention Policy maximum length of time that the Recovery Appliance retains

backups for databases that use this retention policy

Guaranteed Copy guaranteed copy setting, which determines whether backups

protected by this policy must be copied to tape or replicated before being considered

for deletion

Polling Policy optional backup polling policy that determines whether Recovery

Appliance polls a storage location for backups or deletion

Unprotected Window maximum acceptable difference between the current time

and the latest time that the database can be restored

Recovery Window – Index Efficiency (1/2)


Only new version of Data Blocks will be backed up (inc level 1)

Recovery windows controls deletion policy of the blocks

Main scope is to maintain the restorability within the recovery windows

Old and no more needed blocks will be deleted

If enough space is available the Recovery Windows can be overachieved

Source: Oracle

Recovery Window – Index Efficiency (2/2)


Once the old blocks was deleted, we will have a defragmentation within the

storage

This situation will provide random I/O instead of sequential I/O Performance

degradation

Intermittently the RA will automatically reorganize the blocks

Implicitly all touched blocks will be phisical and logical checked on his integrity

Source: Oracle

Storage Locations


Main Storage Location are the Container within the ASM Diskgroup

It is possible to have more than one Storage Location in ASM

Backup Polling Locations are Storage Location outside from ZDLRA (e.g. NFS Mount)

• Backups are placed directly to this location without interacting with ZDLRA

• With Polling Policies you can define how often and where

is the polling directory

• Once all requirement meets e.g. backup related to a

protected database a copy to the RA will be created

• After copying process the protected database will be delete

the backup Automatically from polling directorySource: Oracle


Installation and Configuration

from bottom up

Installation Steps


Create the ZDLRA configuration using

the latest OEDA

Start the Re-Imaging and Setup from the first

Computing Node (like an Exadata Setup)

[root@zdldbadm01 linux-x64]# ./install.sh -cf ./WorkDir/zdl-zdl.xml –r 1-16

Installation Steps


Recovery Appliance specific Installation Steps

[root@zdldbadm01 install]# ./ra_install.pl --help

ra_install.pl - Recovery Appliance Installer

You must be logged in as root to run this command.

All steps should be run successfully for install.

Usage: ./ra_install.pl --help | --step=STEP_NUMBER {--install|--uninstall} [--

stdout]

Options:

--help: displays this help message

--step=number: Which step to run

--stdout: Print all messages to stdout rather then the log file

--install: Installs the step [Default]

--uninstall: Uninstalls the step

Step Numbers:

Step 1 - Validation and Configuration Prep

Step 2 - OS Setup

Step 3 - Oracle User Setup

Step 4 - DBFS Setup

Step 5 - Tape Backup configuration

Step 6 - ZDLRA DB Backup Setup

Step 7 - Enable ZDLRA Services

Configuration Steps


Deployment of the Agents on the compute Nodes

Recovery Appliance Discovery

Installation of the Backup Module on any Clients

[oracle@odax30 zdlra]$ java -jar ra_install.jar -dbUser ra_orcl02 -dbPass

manager -host zdlingest-scan -port 1521 -serviceName zdlra -walletDir

/u01/app/oracle/admin/orcl02/xdb_wallet -libDir $ORACLE_HOME/lib

Recovery Appliance Install Tool, build 2015-11-03

Recovery Appliance credentials are valid.

Recovery Appliance wallet created in directory

/u01/app/oracle/admin/orcl02/xdb_wallet.

Recovery Appliance initialization file

/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/raorcl02.ora created.

Downloading Recovery Appliance Software Library from file ra_linux64.zip.

Downloaded 27189305 bytes in 5 seconds. Transfer rate was 5437861 bytes/second.

Download complete.

[oracle@odax30 zdlra]$


Have a look inside

(just particularities)

On the computing Nodes


Metadata Database

– RAC Database (also used for Recovery Catalog (VPC))

– Non Container Database

– DBFS Mount-Points for Backup and Replication purpose

– DBFS added as Cluster Resources

– HTTP Servlet inside the database will be started for communication between the

clients and the ZDLRA (dbms_ra.startup_recovery_appliance)

oracle@zdldbadm01:~/ [+ASM1] df -h | grep dbfs

[email protected]:/

957M 120K 956M 1% /dbfs_repdbfs

[email protected]:/ 300G 23G 278G 8% /dbfs_obdbfs

SQL> exec dbms_ra.startup_recovery_appliance;

PL/SQL procedure successfully completed.



– Three new FS for internally purposes are provided

[oracle@zdldbadm01 ~]$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/VGExaDb-LVDbSys1

30G 15G 14G 54% /

tmpfs 253G 53M 253G 1% /dev/shm

/dev/sda1 504M 72M 407M 15% /boot

/dev/mapper/VGExaDb-LVDbOra1

99G 59G 36G 63% /u01

/dev/mapper/VGExaDb-LVRADump

296G 191M 281G 1% /radump Dump destination for internal Stuff

/dev/mapper/VGExaDb-LVRABackup

99G 351M 94G 1% /rabackup

/dev/mapper/VGExaDb-LVOBIndex

296G 191M 281G 1% /obindex

[email protected]:/

957M 120K 956M 1% /dbfs_repdbfs

[email protected]:/ 300G 4.6M 300G 1% /dbfs_obdbfs



Backup of Metadata Database

– Done using Export of RASYS Schema every 2 Hours

– Backup Location on DBFS FS: /dbfs_obdbfs/OSB/ra_export

– Scheduling via "anacron"[root@zdldbadm01 ~]# cat /etc/anacrontab

SHELL=/bin/sh

PATH=/sbin:/bin:/usr/sbin:/usr/bin

MAILTO=root

# the maximal random delay added to the base delay of the jobs

#RANDOM_DELAY=45

# the jobs will be started during the following hours only

START_HOURS_RANGE=3-22

#period in days delay in minutes job-identifier command

1 65 cron.daily nice run-parts /etc/cron.daily

7 70 cron.weekly nice run-parts /etc/cron.weekly

30 75 cron.monthly nice run-parts /etc/cron.monthly

[root@zdldbadm01 cron.hourly]# ls -l /etc/cron.hourly/

total 4

-rwx------ 1 root root 409 Nov 10 2015 0anacron

lrwxrwxrwx 1 root root 46 Aug 25 16:20 ra_export.pl -> /opt/oracle.RecoveryAppliance/bin/ra_export.pl



ASM Configuration

– Two Diskgroups DELTA (for Backups), CATALOG (Metadatas)

– New special sub-Directory CONTAINER for storing Container Files which

contains Delta Pools and Delta Stores

– Each Container File as a size of 2 TB ASMCMD [+delta/zdlra] > ls -l

Type Redund Striped Time Sys Name

Y ARCHIVELOG/

Y CONTAINER/

Y DATAFILE/

Y TEMPFILE/

ASMCMD [+delta/zdlra/CONTAINER] > ls -s

Block_Size Blocks Bytes Space Name

512 4294959104 2199019061248 4398059094016 con.365.920823681

512 4294959104 2199019061248 4398059094016 con.366.920823687

512 4294959104 2199019061248 4398059094016 con.367.920823699

...

On the cell Nodes


Fortunatelly nothing special ☺

Same appearance as in Exadata Storage Cell

During the Installation the "Write Back" Option will be enabled automatically for the

Flashcache

Little difference on the Grid Disks compared with the Exadata, only 10 Grid Disks

instead of 12 for the CATALOG ASM Disk Group


From protected Databases

to backups

Protected Databases


Once the Environment meets all the requirements we can add the databases to the

ZDLRA

The requirement are:

• Network connectivity

• Backup Module is installed for every Oracle Home

• Definition of Protection Policies are created (Retention Time)

Block Change Tracking not mandatory, but recommended !!!

Two Methods are possible:

• Enterprise Manager

• Manually DBMS_RA PL/SQL Package

Prerequisite for Adding Protected Database

(EM and manually)


Create an VPD User on the Metadata Database

Grant the Create Session privilege

SQL> create user ra_orcl03 identified by <pwd>;

SQL> grant create session to ra_orcl03;

Adding Protected Database with EM


Add Protected Database on Recovery Appliance Page

– Choose the Protection Policy

Adding Protected Database with EM


Configuring Backup Settings– Register database

– Add RMAN configuration (configure...)

– Enabling Redo Transport

(add parameter to spfile and restart DB)

Scheduling Protected Database with EM


Protected Database is ready for backing up

– We need a Schedule



Configure the repeating interval for

the Incremental-Forever Backups

Configure the Archive Log Backups and

deletion rule



Also useful as Script Generator here the Script is not modifiable !!!



Output of an Backup Schedule on EM13c

Adding Protected Database manually


Add Database for the protected database

Grant access to Virtual Private Catalog

Configure the RMAN Channel for ZDLRA

begin

dbms_ra.add_db (

db_unique_name => 'orcl03',

protection_policy_name => 'bronze',

reserved_space => '250G');

end;

begin

dbms_ra.grant_db_access (

db_unique_name => 'orcl03',

username => 'ra_orcl03');

end;

CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS

"SBT_LIBRARY=/u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libra.so,

ENV=(RA_WALLET='location=file:/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/ra_wallet

credential_alias=zdlingest-scan:1521/zdlra:dedicated')";

Adding Protected Database manually


Enable Real-Time Redo Transport

Restart the Database

ALTER SYSTEM SET REDO_TRANSPORT_USER=ra_orcl03 SCOPE=BOTH;

[oracle@odax30 ~]$ srvctl stop database -db orcl03

[oracle@odax30 ~]$ srvctl start database -db orcl03

Execute Backup manually


Backups works like in traditional way

Only "forever" incremental level 1 backups should be schedule

Level 0 Backup are of course always possible

It works also with old fashion syntax soft link from libra.so to libobk.so is

needed !!!

Scheduling is also free selectable, EM is recommended !

[oracle@odax30 ~]$ rman target / catalog ra_orcl02/manager@zdlingest-scan:1521/zdlra

...

connected to target database: ORCL02 (DBID=3592015831)

connected to recovery catalog database

RMAN> backup incremental level 1 database;

run {

allocate channel c1 type sbt_tape;

backup incremental level 1 database plus archivelog delete input;

}

...

ORA-27211: Failed to load Media Management Library


Copy to Tape

Purpose of Copy to Tape


Tape Infrastructure provides a second or third copy of your backups

Tape Infrastructure could be located on different Data Center DR capability !

Tape backups are "cheaper" for long-term Storage

High efficiency is given in combination with ZDLRA

Oracle Secure Backup just pre-installed on ZDLRA

Setting up Copy to Tape


Configuring Media Management Library• Library Name

• Number of Drives

• Number of Restore Drives

• Media Manager Location (libobk.so)

Setting up Copy to Tape Template


Template helps to define specific attributes

• Backup types

• Database or Protection Policies

• Recovery Windows will be inherited

• Number of copies

• Runtime Window

• Schedule

Executing Copy to Tape


With the scheduling will be iniatiate the copy to tape process

backupset of defined database or protection policy will be copied to tape

Long Term Backup should be done with copy_backup procedure

Restores of backupset from tape will be retrieved transparently !!!


Recovery Capabilities

General statement about Recoveries of protected

Databases


Recoveries of protected databases can be done with EM or manually

With EM several Recovery-Scenarios are possible

All Recovery scenarios with EM are full automated

(e.g. Creation of auxiliary Databases, etc.)

EM generates RMAN Scripts for each Scenario

with the possibility of adaption

Possibility of selection between full and point-in-time

recoveries

Duplicate Database (for standby) need some preparation

(e.g. FS-Structure (admin tree))

Protected Database Recovery with EM


From database Page under Availability Backup & Recovery Perform Recovery



Choose the needed Recovery Scenario

... You can choose an other restore

location if needed...



A schedule will be created

... And you can adapt the script if needed and submit the job...



The Job can be monitored by clicking the "View Job" button...

Protected Database Recovery manually


Restore can be done also manually

– of course db must be mounted before...

– RMAN Client should be configured otherwise parameter are needed (e.g. ENV=...)

Or a point-in-time Recovery

run {

restore database;

recover database;

alter database open;

}

run {

set until time '08.09.2016 08:45:00';

restore database;

recover database;

alter database open resetlogs;

}

Real Time Redo Transport – what happens really ?


We tested what happens really when the protected database crash…

▪ We did a backup of a protected database with enabled real time redo transport

▪ then we shutdown the database inconsistently (abort)… before we checked the actual SCN

▪ the ZDLRA figure out the crash of the protected database and catalog the redo stream from the

staging area as archivelog with the latest SCN he know… that will be latest consistent SCN for

restore and recover the database

RMAN> backup incremental level 1 database plus archivelog delete input;

SQL> select current_scn from v$database;

CURRENT_SCN

-----------

28933241

SQL> shutdown abort

RMAN> list backup of archivelog all;

.

.

1 26 28933197 18-APR-17 28933202 18-APR-17

1 27 28933202 18-APR-17 28933232 18-APR-17

1 28 28933232 18-APR-17 28933246 18-APR-17


Recovery Appliance Views and

Utilities

Some RA Views


It is also possible to read the Recovery Appliance Information over Views from Recover

Appliace Metadata Database

View Name Description

RA_DATABASE Information about Protected Databases

RA_DB_ACCESS Describes User Accounts that have access to which protected database

RA_PROTECTION_POLICY Protection Policies defined on the Recovery Appliance

RA_DATABASE_STORAGE_USAGE Storage usage for each protected database

RA_RESTORE_RANGE Describe Restore Range for each protected database

RA_INCIDENT_LOG Describe Recovery Appliance Incidents

RA_REPLICATION_SERVER Replication Server Information (if used)

rastat.pl Utility


rastat Utility help to evaluate the performance of Recovery Appliance

– NETBACKUP/NETRESTORE

• Measure the network performance

– ASMREAD/ASMWRITE

• Measure the ASM Disk I/O Performance

– CONTAINERREAD/CONTAINERWRITE/CONTAINERALLOC

• Measure the Container Disk I/O Performance

[oracle@zdldbadm01 ~]$ perl /opt/oracle.RecoveryAppliance/client/rastat.pl --test=CONTAINERWRITE --

filesize=2048M --rasys=rasys/manager@zdlingest-scan:1521/zdlra --diskgroup=/:DELTA

RECOVERY APPLIANCE WRITE IO TEST TO CONTAINER FILES

Disk Group: /:DELTA

2048 MB, .75s write IO time, .51s CPU time, 2724.80 MB/s, 68.51% CPU usage

PL/SQL procedure successfully completed.

dbms_ra Package


dbms_ra provides administration function from inside the RA Database

Ca. 50 Procedure for different types of administration

– Start/Stop of Recovery Appliance

– Add/Delete Protected Databases

– Grants Access to specific User to enable to backup and restore

– Add/Update Protection Policies

– Manage Tape Configuration (if available)

– Add Repliaction Server (if available)

But, the utilization of Enterprise Manager is recommended !!!


Conclusion

Conclusion


ZDLRA worked in our Tests as

expected

Very good implementation of well

known Backup and Recovery

Technology on Engineered System

RPO is (near) zero with Real Time

Redo Transport

Even RTO ist much better then

existing systems

Very simple Management with EM

Risk reduced, cost reduced,

qualitiy improved

ZDLRA will find it’s

customer !

Questions and Answers...

Daniele Massimi – Principal Consultant

Tel. +41 58 459 50 92

[email protected]

21.06.2017 ZDLRA - in Action63

zdlra deep dive

Documents