the clone task force - dell emceducation.emc.com/.../clone_task_force_article_liso.pdfthe clone task...

© Copyright 2007 EMC Corporation 1

The Clone Task Force Local Replication, CLARiiON® Environment

EMC Proven® Professional Knowledge Sharing May, 2007

Fernando Moreno Liso System Executive Engineer � CLARiiON Specialist

Comparex Espana, S.A


Table of Contents

AUTHOR�S BIOGRAPHY ..............................................................................................................................3

ABSTRACT ..........................................................................................................................................................4

ARTICLE: THE CLONE TASK FORCE......................................................................................................4

1. PLANNING THE CLONES .......................................................................................................................5

CASE 1. AIX 5.3, ORACLE 10..............................................................................................................................6 CASE 2. SLES 8, ORACLE 9I...............................................................................................................................7

2. PRE-REQUISITES......................................................................................................................................8

2.1 NAVICLI .........................................................................................................................................................8 2.2 JAVA ...............................................................................................................................................................8 2.3 SECURITY.......................................................................................................................................................8

3. SYNCHRONIZING .....................................................................................................................................9

AIX ..................................................................................................................................... 9 Linux ............................................................................................................................... 10

4. FRACTURING.............................................................................................................................................12

4.1 CONSISTENCY ..............................................................................................................................................12 4.2 INTEGRITY WITH ORACLE 10 .......................................................................................................................14 4.3 INTEGRITY WITH ORACLE 9I.........................................................................................................................15

5. BACKING UP ..............................................................................................................................................16

6. DEVELOPING AND TESTING.............................................................................................................18

7. CONCLUSIONS .........................................................................................................................................18


Author�s Biography Fernando was born in Spain, and is married with a son and a daughter. After completing his degree, he was a systems engineer with HP Net servers learning about IT architecture. Some years later, as a system administrator at the same company specializing in Microsoft environments, he learned about the �soft� side of IT architectures. At Comparex, he focuses on storage subsystems. He specialized in HP and EMC platforms achieving HP Master Accredited Systems Engineer and EMC Proven Professional Customer Engineer (CLARiiON Specialist) status. Fernando now works as a storage services consultant to improve operational facilities and data availability in IT departments. Disclaimer: The views, processes or methodologies published in this compilation are those of the authors. They do not necessarily reflect EMC Corporation�s views, processes or methodologies.


Abstract Designing: Needs and Challenge. Data availability and protection must be considered when designing and implementing new storage information architectures. Since most systems are in production when these architectures are implemented, downtime must be avoided while still performing availability task routines. Backups can also fail so the testing of processes under crash conditions is also paramount. The work described in this article, based upon EMC best practices, describes the challenge of implementing local replication, backup, restore, and recovery of AIX and Linux (SUSE Linux Enterprise Server � or SLES for short) environments with Oracle and SAP applications through the scheduling of automatic tasks that involve cloning, backup and re-charge of test systems� production databases. Our project was a success. Through the use of scheduling tools, navicli commands and database scripting, the replica execution process was vastly simplified. In fact, only one script was required to recover an entire database twice a day. Implementing the method. The process was simple. We synchronized and fractured three groups of clones automatically for each environment (AIX or Linux, SLES) every eight hours, and assigned one of the fractured clone groups to a backup server. Then, we restored to a test server and recovered and checked the data (SAP and Oracle databases) within another instance name. Although the process was simple, the method was difficult to develop. We had to consider the implementation and integration of several tools in sequence including crontab, navicli commands, and SAP and Oracle scripts. We had to suspend different versions of databases, sync, consistent fracture, resume database, assign clones, backup to a disk library, de-assign clones, restore data, and start instances in test servers� all without impacting production systems. Additionally, all these processes needed to run reliably and automatically with only one click.

Article: The Clone Task Force There are some special requirements to make CLARiiON�s local replication and high availability easy and effortless to manage. The objective of this article is to describe the local replication implemented in our main production environment. This implementation was necessary to protect the information system�s data. We wanted to protect data every eight hours and to retain copies for as much time as possible (23 hours) so we implemented a regular system for cloning the information without impacting performance. The CLARiiON offered many advantages.


We scheduled fractures every day as follows:

• Fractured first clones at 16:00; synchronized them at 15:00.

• Fractured second clones at 00:00; synchronized them at 23:00

• Fractured third clones at 08:00; synchronized them at 07:00.

3 Different Fractures every 8h and kept 23h.

We also wanted to give developers an opportunity to test with recent production data. Administrators required the same opportunity to backup data up to 23 hours. One of those fractured clones was assigned to a host for testing and backup tasks and then replaced to the cloning rotation once a day. The custom environment described in this article is the base infrastructure to implement local replication in CLARiiON. It consists of the following: - 1 CLARiiON CX700 array (Release 19). - Navisphere® Manager, Agent and CLI, SnapView and PowerPath® Soft. - Logical Units (LUN) over FC disks assigned to production hosts. - The same 3n of LUN for each clone as required, the same size, over ATA disks for

cloning purposes. - 2 LUN of 0,125GB for clone private feature properties configured. - 1 host of each OS to mount the clones (AIX and SLES). - 1 CLARiiON Disk Library for backing up tasks.

In addition, the customer environment included SAP and Oracle databases on the production LUN. We had to know Oracle and SAP commands to make the replication consistent and to have the instances up and running in the background or development environments.

1. Planning the Clones We employed two ways of cloning. - The first consisted of an SAP R/3 environment containing production IS management

system under Oracle 10 database and AIX 5.3 OS in a Power5 partitioned server. - The second consisted of another SAP environment containing HR and FI

management systems under Oracle 9.2 databases and SLES 8 Linux Enterprise at a Blade center.


The storage design of those databases follows:

Case 1. AIX 5.3, Oracle 10 An IBM cluster (HACMP 5.2) with two AIX 5.3 nodes held the following storage infrastructure, all supporting the PA1 Information Data System:

SAP2SAP1

OriglogLUN50 2,5GB Raid1/0

sapdataLUN10 270GB Raid5

InstanciaLUN20 20GB

Raid1 MirrlogLUN40 2,5GB Raid1/0

archivesLUN30 10GB Raid5

We built the following infrastructure on the CX700 to maintain the clone replication:

FC DisksSource LUN Comments Size (GB) Raid Clone Group Clone1 Clone2 Clone3

LUN10 Sapdatas (1-5) 270 5 clones_pa1_lun10 LUN1010 LUN1011 LUN1012LUN20 Instance 20 1 clones_pa1_lun20 LUN1020 LUN1021 LUN1022LUN30 Archives/Redo Logs Offline 10 5 clones_pa1_lun30 LUN1030 LUN1031 LUN1058LUN40 Redo Logs Online (OrigLog A&B) 2,5 1/0 clones_pa1_lun40 LUN1040 LUN1041 LUN1042LUN50 Mirrored Logs Online (MirrLog A&B) 2,5 1/0 clones_pa1_lun50 LUN1050 LUN1051 LUN1052

SAP IS PA1 - AIX 5.3, Oracle 10ATA Disks Raid5

We prepared the AIX and Oracle scripts for synchronizing and fracturing these clones regularly and consistently.


Case 2. SLES 8, Oracle 9i A heartbeat SLES8 Linux cluster with two nodes sharing the (PH2) SAP Eco-Fin and HR information data system:

Dbpro2bDbpro2a

Sapdatas1-4LUN60 200GB Raid5

Sapdatas5-8LUN70 300GB Raid5

LogsLUN80 20GB Raid1/0

Also, to maintain the clone replication, we built the following infrastructure on the CX700:

FC DisksSource LUN Comments Size (GB) Raid Clone Group Clone1 Clone2 Clone3

LUN60 Sapdatas (1-4) 200 5 clones_ph2_lun60 LUN1060 LUN1061 LUN1062LUN70 Sapdatas (5-8) 300 5 clones_ph2_lun70 LUN1070 LUN1071 LUN1072LUN80 Logs Online & Offline 20 1/0 clones_ph2_lun80 LUN1080 LUN1081 LUN1082

ATA Disks Raid5

SAP HR&FI PH2 - SLES8, Oracle 9

Then, we showed the synchronizing and fracturing procedures periodically and consistently.


2. Pre-Requisites Here are some considerations before initiating the cloning process.

2.1 NaviCLI As long as Release 19 is implemented in the CX, we needed the supported navicli version to interact with the array. For both, AIX and Linux, the installed navicli version is 6.19.2 (6.5) navicli getagent Agent Rev: 6.19.2 (6.5)

2.2 Java We had to install Java 1.4.2 for users who would be executing the scripts for synchronizing and fracturing the clones because the Navisphere Command Line Interface (CLI) for clones uses Java runtime, and because Java was already implemented for SAP in earlier versions than required or recommended for cloning CLI. For AIX: java -jar -version java version "1.4.2" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2) Classic VM (build 1.4.2, J2RE 1.4.2 IBM AIX 5L for PowerPC (64 bit JVM) build cAIX64142-20060824 (SR6) (JIT enabled: jitc)) The installation directory for this version is: /usr/java/j2re1.4.2_12/bin/ For Linux: java -jar -version java version "1.4.2_12" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_12-b03) Java HotSpot(TM) Client VM (build 1.4.2_12-b03, mixed mode) Installation directory for this version is: /usr/java/j2re1.4.2_12/bin

2.3 Security The user who launched the synchronization and fracture scripts required privileges at the CX. We had to implement the CLI key and CLI security file to work around this issue. navicli.jar -AddUserSecurity -password *password* -scope 0


3. Synchronizing Synchronization occurs one hour before the fracture of each group of clones in each environment. The first synchronization lasted longer than one hour but after this, we experienced medium rates (a high rate could impact performance) and our results were adequate an hour before fracturing. When synchronizing a clone, we lost the availability to recover from the time it was fractured so we did have not a retention backup each 8 hours of 24 but rather 23 hours. That was the time we kept a clone copy of the data, three copies a day. The archive logs did the rest. This is the synchronization process scheduled in the production hosts:

AIX crontab �l # SNAPCLONE 15:00, SINC CLONE1 0 15 * * * /root/snapclone1500 > /tmp/snapclone1500.log # SNAPCLONE 23:00, SINC CLONE2 0 23 * * * /root/snapclone2300 > /tmp/snapclone2300.log # SNAPCLONE 07:00, SINC CLONE3 0 7 * * * /root/snapclone0700 > /tmp/snapclone0700.log Each of those scripts contains the following: :/root $cat snapclone1500 echo "######################################" echo Fecha: date +"%Y%m%d%H%M%S" echo Sincro Clones1 de PA1 ... java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun10 -cloneid 0100000000000000 -o java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun20 -cloneid 0100000000000000 -o java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun30 -cloneid 0100000000000000 -o java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun40 -cloneid 0100000000000000 -o java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun50 -cloneid 0100000000000000 -o echo "######################################" echo "EXIT --- FIN DEL SCRIPT --- CLONEs1 PA1 Sincronizados" exit


:/root $cat snapclone2300 echo "######################################" echo Fecha: date +"%Y%m%d%H%M%S" echo Sincro Clones2 de PA1 ... java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun10 -cloneid 0200000000000000 -o java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun20 -cloneid 0200000000000000 -o java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun30 -cloneid 0200000000000000 -o java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun40 -cloneid 0200000000000000 -o java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun50 -cloneid 0200000000000000 -o echo "######################################" echo "EXIT --- FIN DEL SCRIPT --- CLONEs2 PA1 Sincronizados" exit :/root $cat snapclone0700 echo "######################################" echo Fecha: date +"%Y%m%d%H%M%S" echo Sincro Clones3 de PA1 ... java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun10 -cloneid 0300000000000000 -o java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun20 -cloneid 0300000000000000 -o java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun30 -cloneid 0300000000000000 -o java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun40 -cloneid 0300000000000000 -o java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_pa1_lun50 -cloneid 0300000000000000 -o echo "######################################" echo "EXIT --- FIN DEL SCRIPT --- CLONEs3 PA1 Sincronizados" exit

Linux crontab �l # SNAPCLONE 15:00, SINC CLONE1 0 15 * * * /root/scripts/snapclone1500 > /root/scripts/snapclone1500.log # SNAPCLONE 23:00, SINC CLONE2 0 23 * * * /root/scripts/snapclone2300 > /root/scripts/snapclone2300.log # SNAPCLONE 07:00, SINC CLONE3 0 7 * * * /root/scripts/snapclone0700 > /root/scripts/snapclone0700.log


And the scripts called from crontab: :/root/scripts $cat snapclone1500 echo "######################################" echo Fecha: date +"%Y%m%d%H%M%S" echo Sinc de Clones1 PH2 ... /usr/java/j2re1.4.2_12/bin/java -jar /opt/Navisphere/bin/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_ph2_lun60 -cloneid 0100000000000000 -o /usr/java/j2re1.4.2_12/bin/java -jar /opt/Navisphere/bin/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_ph2_lun70 -cloneid 0100000000000000 -o /usr/java/j2re1.4.2_12/bin/java -jar /opt/Navisphere/bin/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_ph2_lun80 -cloneid 0100000000000000 -o echo "######################################" echo "EXIT --- FIN DEL SCRIPT --- CLONEs1 PH2 Sincronizados" exit :/root/scripts $cat snapclone2300 echo "######################################" echo Fecha: date +"%Y%m%d%H%M%S" echo Sinc de Clones2 PH2 ... /usr/java/j2re1.4.2_12/bin/java -jar /opt/Navisphere/bin/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_ph2_lun60 -cloneid 0200000000000000 -o /usr/java/j2re1.4.2_12/bin/java -jar /opt/Navisphere/bin/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_ph2_lun70 -cloneid 0200000000000000 -o /usr/java/j2re1.4.2_12/bin/java -jar /opt/Navisphere/bin/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_ph2_lun80 -cloneid 0200000000000000 -o echo "######################################" echo "EXIT --- FIN DEL SCRIPT --- CLONEs2 PH2 Sincronizados" exit :/root/scripts $cat snapclone0700 echo "######################################" echo Fecha: date +"%Y%m%d%H%M%S" echo Sinc de Clones3 PH2 ... /usr/java/j2re1.4.2_12/bin/java -jar /opt/Navisphere/bin/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_ph2_lun60 -cloneid 0300000000000000 -o /usr/java/j2re1.4.2_12/bin/java -jar /opt/Navisphere/bin/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_ph2_lun70 -cloneid 0300000000000000 -o /usr/java/j2re1.4.2_12/bin/java -jar /opt/Navisphere/bin/navicli.jar -address 10.165.4.72 snapview -syncclone -name clones_ph2_lun80 -cloneid 0300000000000000 -o echo "######################################" echo "EXIT --- FIN DEL SCRIPT --- CLONEs3 PH2 Sincronizados" exit


4. Fracturing When we fractured a clone group, we considered the Oracle databases� consistency and integrity. We had to bind or enchain some tasks in our scheduled scripts.

4.1 Consistency When fracturing more than one clone simultaneously, and to preserve the point in time across the set of clones, we used the -consistentfractureclones option in our scripts. In this way, we ensured the I/O requests would delay until all of the clones had been fractured. Fractures occurred three times a day, after synchronization of the respective clones one hour before. Synchronization of the file systems was also needed to flush cache data into a disk. Fractures occurred at 00:00, 08:00 and 16:00 when clones were already in a synchronized or consistent state.

Main fracture scripts are called by crontab on each system. AIX crontab �l # SNAPCLONE 16:00, FRACT CLONE1 0 16 * * * /root/snapclone1600 > /tmp/snapclone1600.log # SNAPCLONE 00:00, FRACT CLONE2 0 0 * * * /root/snapclone0000 > /tmp/snapclone0000.log # SNAPCLONE 08:00, FRACT CLONE3 0 8 * * * /root/snapclone0800 > /tmp/snapclone0800.log Additional scripts follow on the next page.


Each of these scripts contains lines for each fracture: :/root $cat snapclone1600 echo "######################################" echo Fecha: date +"%Y%m%d%H%M%S" rm /tmp/bckctrl.dbf echo "######################################" echo Suspend BBDD ... su - orapa1 "-c sqlplus /nolog @'/root/snapsuspend.sql'" echo "######################################" echo Limpieza de buffers y Sincronizacion datos filesystems en cache ... /usr/sbin/sync /usr/sbin/sync echo Fractura Consistente de Clones1 de PA1 java -jar /usr/lpp/NAVICLI/navicli.jar -address 10.165.4.72 snapview -consistentfractureclones -CloneGroupNameCloneId clones_pa1_lun10 0100000000000000 clones_pa1_lun20 0100000000000000 clones_pa1_lun30 0100000000000000 clones_pa1_lun40 0100000000000000 clones_pa1_lun50 0100000000000000 -o echo "######################################" echo Resume BBDD ... su - orapa1 "-c sqlplus /nolog @'/root/snapresume.sql'" echo "######################################" echo "EXIT --- FIN DEL SCRIPT --- CLONEs2 PA1 fracturados" exit And the same at 00:00, 08:00 for second (0200000000000000) and third (0300000000000000) snap clones respectively. sanpsuspend.sql and snapresume.sql scripts will be discussed in the integrity section below. Linux crontab �l # SNAPCLONE 16:00, FRACT CLONE1 0 16 * * * /root/scripts/snapclone1600 > /root/scripts/snapclone1600.log # SNAPCLONE 00:00, FRACT CLONE2 0 0 * * * /root/scripts/snapclone0000 > /root/scripts/snapclone0000.log # SNAPCLONE 08:00, FRACT CLONE3 0 8 * * * /root/scripts/snapclone0800 > /root/scripts/snapclone0800.log Additional scripts follow on the next page.


One of these scripts is implemented as follows: Linux:/root/scripts $cat snapclone0000 echo "######################################" echo Fecha: date +"%Y%m%d%H%M%S" echo "######################################" echo Suspend BBDD ... su - oraph2 "-c rm /tmp/bckctrl.dbf" su - oraph2 "-c sqlplus /nolog @'snapsuspend.sql'" echo "######################################" echo Limpieza de buffers y Sincronizacion datos filesystems en cache ... /bin/sync /bin/sync echo "######################################" echo Fractura de Clones2 de PH2 /usr/java/j2re1.4.2_12/bin/java -jar /opt/Navisphere/bin/navicli.jar -address 10.165.4.72 snapview -consistentfractureclones -CloneGroupNameCloneId clones_ph2_lun60 0200000000000000 clones_ph2_lun70 0200000000000000 clones_ph2_lun80 0200000000000000 -o echo "######################################" echo Resume BBDD ... su - oraph2 "-c sqlplus /nolog @'snapresume.sql'" echo "######################################" echo "EXIT --- FIN DEL SCRIPT --- CLONEs2 PH2 fracturado" exit We changed the clone id to fracture for the first and third fracture clones scripts. Inside these scripts we introduced other commands (snapsuspend and snapresume) to put the Oracle database in special mode, and to give integrity to the cloned database.

4.2 Integrity with Oracle 10 We achieved Oracle database integrity by freezing (suspend mode) the instance we wanted to backup through cloning tasks. It involved the following:

• Begin backup of Table spaces • Suspend database

It is easier with Oracle release 10 since �alter database begin backup� allowed putting the whole tablespace in begin backup mode. Then, we put put the database in suspend mode. Additional scripts follow on next page.


:/root $cat snapsuspend.sql connect / as sysdba !echo database backup controlfile to trace; alter database backup controlfile to trace; !echo database backup controlfile to file tmp bckctrl; alter database backup controlfile to '/tmp/bckctrl.dbf'; !echo system switch logfile; alter system switch logfile; !echo database begin backup; alter database begin backup; !echo SYSTEM SUSPEND; alter system suspend; !echo exit; exit :/root $cat snapresume.sql connect / as sysdba alter system resume; alter database end backup; alter system archive log current; alter database backup controlfile to trace; alter database backup controlfile to '/tmp/bckctrl.dbf'; exit

4.3 Integrity with Oracle 9i For Oracle�s 9.2 release you must consider table space. To build the Oracle script you can list them with this command: select to_char(sysdate, 'MM/DD/YY') datum, tablespace_name, status from sys.dba_tablespaces; Snapclone calls snapsuspend.sql which includes these statements: Linux:/root/scripts $cat snapsuspend.sql connect / as sysdba alter database backup controlfile to trace; alter system switch logfile; !echo TABLESPACES BEGIN BACKUP; start beginbackup.sql !echo ALTER SYSTEM SUSPEND; alter system suspend; exit


beginbackup.sql contains the list of sql statements to put all the tablespaces previously listed into begin backup mode: Linux:/root/scripts $cat beginbackup.sql !echo TABLESPACE SYSTEM begin backup; alter tablespace SYSTEM begin backup; !echo TABLESPACE PSAPROLL begin backup; alter tablespace PSAPROLL begin backup; ��. And so on until the last tablespace. When the clone set is already fractured, snapclone calls snapresume.sql which calls endbackup.sql to complete the process, where: Linux:/root/scripts $cat snapresume.sql connect / as sysdba alter system resume; start endbackup.sql alter system archive log current; alter database backup controlfile to trace; alter database backup controlfile to '/tmp/bckctrl.dbf'; exit And: Linux:/root/scripts $cat endbackup.sql alter tablespace SYSTEM end backup; alter tablespace PSAPROLL end backup; ��. And so on until the last tablespace. The last two scripts must reside in the Oracle user home directory with the appropriate permissions.

5. Backing Up We assigned clones to backup hosts through the Veritas pre-script �bpstart_notify.policy� executed just after one of the fractures: navicli -h 10.165.4.72 storagegroup -addhlu -gname SG_AIXBACKUP -hlu 0 -alu 1010 navicli -h 10.165.4.72 storagegroup -addhlu -gname SG_AIXBACKUP -hlu 1 -alu 1020 navicli -h 10.165.4.72 storagegroup -addhlu -gname SG_AIXBACKUP -hlu 2 -alu 1030 navicli -h 10.165.4.72 storagegroup -addhlu -gname SG_AIXBACKUP -hlu 3 -alu 1040 navicli -h 10.165.4.72 storagegroup -addhlu -gname SG_AIXBACKUP -hlu 4 -alu 1050 cfgmgr powermt config rc.agent start varyonvg vg01 varyonvg vg02 varyonvg vg03 varyonvg vg04 varyonvg vg05 sh /root/scripts/PA1mounts.sh


PA1mounts.sh included the mount command of each of the file systems required for backup (SAPdatas and logs) and defined in /etc/filesystems (Aix) or /etc/fstab (Linux). The backup put the clone information on a CDL as if it were on a tape for longer retention and without impacting production hosts. In Linux, we scheduled a reboot after assigning the clones to the backup host. It is the only way we have found to effectively rescan the scsi channel with SLES8. So, we scheduled the activation of clones and the reboot through crontab. 30 15 * * * /root/scripts/snapActivarCLONEs1 > /tmp/snapActivarCLONEs1.log 35 15 * * * /sbin/shutdown -r now The rest of the script is the same as in AIX but we activated volume groups with vgchange �a y vgname and we flushed file systems cache with /bin/sync Once backup was completed, we had to de-assign clones and remove any clone disk reference for subsequent assignment the next day. You can do it automatically by including these lines in your post backup script (bpend_notify.policy in case of Veritas Netbackup for the example we use). For AIX, assuming hdisk0 and hdisk1 are internal disks and hdisk3 to hdisk12 are the 10 paths for the 5 disk clones: #!/bin/ksh /usr/sbin/sync /usr/sbin/sync # Umount all the file systems sh /root/SCRIPTS/PA1umounts.sh # stop I/O rc.agent stop # De-activate VGs Vg=�vg01 vg02 vg03 vg04 vg05� For volume in $Vg Varyoffvg $volume done # Remove PowerPath devices powermt remove dev=all lsdev -Cc disk -Fname | grep power | xargs -n1 rmdev -dl rmdev -dl powerpath0 hdisktoremove="hdisk3 hdis4 hdisk5 hdisk6 hdisk7 hdisk8 hdisk9 hdisk10 hdisk11 hdisk12" for r in $hdisktoremove do rmdev -dl $r done rc.agent start sleep 1 # De-assign clones. navicli -h 10.165.4.72 storagegroup -removehlu -gname JARA_RUMPUP -hlu 0 -o navicli -h 10.165.4.72 storagegroup -removehlu -gname JARA_RUMPUP -hlu 1 -o navicli -h 10.165.4.72 storagegroup -removehlu -gname JARA_RUMPUP -hlu 2 -o navicli -h 10.165.4.72 storagegroup -removehlu -gname JARA_RUMPUP -hlu 3 -o navicli -h 10.165.4.72 storagegroup -removehlu -gname JARA_RUMPUP -hlu 4 -o


It was not necessary to remove PowerPath devices for Linux hosts. We kept the paths in a dead state since they would be the same pseudo devices after re-assigning and rebooting.

6. Developing and Testing We tested the reliability of the replication in two ways:

• Recovering from one of the fractured clones • Restoring from CDL virtual tape backup

In both AIX and Linux, we assigned (the same way we assigned for backing up) one of the fractured clones to the test hosts and recovered a new database instance with the same data information weekly, or we restored one of the fractured clones from tape. To automate this task, you must: • Remember that the name of the database instance should be different than the original. • Change the Tnsnames and Listener files according to new host. • Change the SAP profile to accomplish new host. • Secure an SAP license for the new hardware machine to bring up the test or develop

environment. • Change the Oracle user according to new test environment. • Perform Recover Database to run the new database. • Modify or disable R3 jobs of source instance.

7. Conclusions Once implemented, the cloning procedure we see as cron daemon for UNIX servers provided the main functionality to schedule scripts that called another one to fully automate the process. It was not difficult to achieve a replication system for our data in conjunction with CLI and Oracle concepts. We recommend that you monitor the progression of synchronization and fracture to self-control the course of action. We also recommend that you get statistics from analyzer to minimize any possible impact on performance if more than one clone synchronization could be on. From SAP systems, RZ20, ST03 or SM50 would help to monitor the system.

the clone task force - dell emceducation.emc.com/.../clone_task_force_article_liso.pdfthe clone task...

Documents