disaster recovery plan

CASSIS Wave 1 Disaster Recovery Plan

CASSIS Wave 1R/3 Disaster Recovery Plan

DOCUMENT NAME & LOCATION: C1_BAI_DRP

TBD

VERSION: 0.01

DATE: 11/21/05

READERSHIP: C1 Project

SUMMARY: This is the Disaster Recovery Plan (DRP) for C1 Production R/3 PRE.

Associated Documents (this document should be read in conjunction with)Title of Document File Location (refer to the most recent version of the

file)

Technical Infrastructure Plan

Operations Manual


Document Information

Document History

Version Author(s) Comments/Changes Revision Date0.01 Anna Wick First Draft 11/21/05

Document Approval

Approver's Name Department/Group Date Approved

Document Distribution

Name or Group Location Date Distributed

Review (e.g., SQA Review, Peer Review)

Review Date Comments Approval Signature

of 12


Table of Contents

1. Purpose of this Document.......................................................................................................5

2. Approach................................................................................................................................ 5

2.1 Plan Summary and Objectives................................................................................................5

2.2 Design Assumptions...............................................................................................................5

2.3 Process Assumptions.............................................................................................................5

2.4 Notification Contacts...............................................................................................................5

3. SAP R/3 PRE Disaster Recovery Procedures.........................................................................6

3.1 <Basis> Stop the PFE system and Database..........................................................................6

3.1.1 Stop Backup if Running......................................................................................................63.1.2 Shutdown SAP & Oracle Applications.................................................................................63.1.3 Stop Oracle Services..........................................................................................................63.1.4 Stop SAP Services.............................................................................................................63.2 <Basis> Remove old PFE files and rename PFE directories to PRE........................................6

3.2.1 Remove files...................................................................................................................... 63.2.2 Rename directories............................................................................................................63.3 <Basis> Central Instance & Application Server Installation......................................................6

3.4 <Basis> Prepare to restore PRE database to PFE..................................................................7

3.4.1 Set the environment variables for PREadm.........................................................................73.4.2 Create the initPRE.utl file....................................................................................................73.4.3 Regenerate BKI file............................................................................................................83.4.4 Obtain the backPRE.log file from tape................................................................................83.4.5 Obtain the most recent backup’s logfile from tape...............................................................83.4.6 Obtain the initPRE.ora parameter file from tape:.................................................................83.4.7 Obtain the initPRE.sap parameter file from tape:.................................................................83.4.8 Obtain the initPRE.dba parameter file from tape:................................................................83.5 <Basis> Restore PRE Files from Tape....................................................................................8

3.5.1 Restore datafiles................................................................................................................83.5.2 Restore Archive logs..........................................................................................................83.6 <Basis> Recover PRE............................................................................................................9

3.6.1 Start Oracle Service for PRE..............................................................................................93.6.2 Roll forward database until last log......................................................................................93.7 <Basis> Open PRE database.................................................................................................9

3.8 <Basis & IBM Tulsa> Update /usr/sap/trans............................................................................9

3.8.1 <Basis> Create /usr/sap/trans directory..............................................................................93.8.2 <IBM Tulsa> Restore /usr/sap/trans...................................................................................93.9 <Basis> Configure Oracle Files and Start Listener..................................................................9

3.10 <Basis & IBM Tulsa> Initial Start & SAP Configuration......................................................10

3.10.1 <IBM Tulsa> Update DNS............................................................................................103.10.2 <Basis> Request SAP license......................................................................................103.10.3 <Basis> Install New SAP License.................................................................................10

of 12


3.10.4 <Basis> Restore SAP profiles from tape.......................................................................103.10.5 <Basis> Update SAP profiles as needed.......................................................................103.10.6 <Basis> Remove Batch and Spool Processes from the system.....................................103.10.7 <Basis> Start PRE Central Instance.............................................................................103.10.8 <Basis> Lock All users.................................................................................................103.10.9 <Basis> Import instance profiles...................................................................................103.10.10 <Basis> Operation Modes............................................................................................103.10.11 <Basis> Execute the initial SAP consistency check.......................................................103.10.12 <Basis> Check Partner profile in PRE..........................................................................103.10.13 <Basis> RFC Server Groups........................................................................................113.11 <Basis> Final Start of PRE with Batch..............................................................................11

3.11.1 To Activate the batch and spool processes...................................................................113.11.2 Resume Batch Scheduling............................................................................................113.11.3 Start PRE.....................................................................................................................113.11.4 Setup Login Groups......................................................................................................123.12 <Basis> Final Steps..........................................................................................................12

3.12.1 Resume Operations procedures...................................................................................123.12.2 Configure STMS...........................................................................................................12

of 12


1. PURPOSE OF THIS DOCUMENT

This document outlines the Disaster Recovery Plan (DRP) for the CASSIS Wave 1 (C1) R/3 Production instance, PRE. In general, PRE will come up on the PFE hardware at the BRDC in case of Disaster. This document assumes that the database / central instance server for PFE already exist and are set up from an infrastructure standpoint.

2. APPROACH

2.1 Plan Summary and ObjectivesThis document provides a detailed plan for the recovery of the C1 R/3 Production environment in the event a disaster is declared at the Tulsa Data Center (TDC) located in Tulsa, Oklahoma. The system covered is PRE SAP R/3 (database / central instance server & application server).

2.2 Design AssumptionsThe following assumptions are made regarding the SAP disaster recovery procedures:

The Primary Site, the Tulsa Data Center (TDC), is located at 509 S. Boston Ave, Tulsa, OK 74103-4602, and holds the production servers for SAP R/3 PRE.

Disaster recovery site will be the Backup Recovery Data Center (BRDC) site, located approximately 8 miles from the TDC at CityPlex, 2488 East 81st, Suite 455, Tulsa, OK 74137.

The BRDC is used for development and testing and serves as the disaster recovery site in the event of a disaster.

Servers in the BRDC will already host the executables, and interface executables, that exist on the production servers

In the event of a disaster, the production fix (PFE) environment will be used to house production. Original PFE will not be available until either a return of the PRE environment to original production hardware, or additional hardware is procured to replace the PFE environment.

Network throughput is sufficient to handle traffic between TDC and BRDC. Network routing is in place such that, if the TDC is destroyed, external users could be routed through

other pre-established locations (e.g. Omaha). All communication to the PRE SAP R/3 and other associated servers is done via DNS. It is assumed

that changing the DNS entry to point to the new locations after disaster will allow all satellite systems to find the environments.

This plan does not cover rolling disasters. That is, if the BRDC was hit by disaster at the same time or soon after the TDC was declared under disaster, this plan does not handle that recovery.

This plan does not cover loss of the BRDC. If the BRDC were unavailable, there would be no recovery site for the TDC and development processes would cease until the BRDC is restored.

2.3 Process Assumptions A disaster has been declared at the TDC site.

2.4 Notification ContactsOnce the disaster has been declared, the following process should be followed to notify Application personnel and begin the “Disaster Recovery Procedures” indicated in this document. The BP Disaster Declaration team will contact the following people to notify them of a disaster.

Overall Contacts to start notification process: TBD HW/OS: IBM Hosting Windows On-Call Networking: IBM Hosting Windows On-Call DBA/Basis: TBD Batch Scheduling: TBD Backup/Recovery: Basis On-Call, IBM Hosting Windows On-Call Business Contact (for recovery completion): TBD

of 12


Once all parties have been contacted, the appropriate group will perform the procedures in their sections in the order noted below. When each group in succession has completed their step and verified the steps, they will notify the hand-off organization that will then begin their respective steps.

3. SAP R/3 PRE DISASTER RECOVERY PROCEDURESThis section provides the process that would be followed in the event that a disaster is declared for the PRE Environment, including the database server and central instance server and/or associated storage.

3.1 <Basis> Stop the PFE system and Database3.1.1 Stop Backup if Running

Check if there is a PFE database backup running, if there is, stop the backup now.3.1.2 Shutdown SAP & Oracle Applications

Use SAP Management console to stop SAP Use SAPDBA to shutdown Oracle

3.1.3 Stop Oracle ServicesStop the following Oracle Services:

OracleOraHome92Agent OracleOraHome92TNSListener OracleServicePFE

3.1.4 Stop SAP ServicesStop the following SAP Services

SAPOSCOL SAPPFE_10

3.2 <Basis> Remove old PFE files and rename PFE directories to PRE3.2.1 Remove files

Remove all datafiles under the sapdata directories on each drive where datafiles exist Do not remove the sapdata directoriesRemove origlog and mirrorlog files Do not remove origlog* and mirrlog*directories Do not remove the cntrl directory under the origlog* and mirrlog* directories

3.2.2 Rename directories Rename all <Drive>:\oracle\PFE to <Drive>:\oracle\PRE

3.3 <Basis> Central Instance & Application Server InstallationFollow the steps below to install SAP on the Central Instance (BP1TULAP113) for PRE. The database host is BP1TULAP192.1. Start Sapinst on the host:

Start -> Programs -> SAPInst Oracle Kernel_PRE -> select SAPinst Server+Gui2. Welcome window

Select SAP R/3 Oracle – Default Click on Next

3. License agreement Click on I Agree

4. SAP System Common Parameters SAP System ID: PRE Select Central Instance Click on Next

5. SAP System Common Parameters Instance Number: 10 Instance Host: <BP1TULAP113.amoco.com> Database System ID: PRE Database Host: <BP1TULAP192.amoco.com> Click on Next

6. SAP Instance Host Information Choose Local Installation

of 12


Ram:<8GB> Click on Next

7. SAP System Instance Directories Choose the <DRIVE>: for SAP system directory Host with Transport Directory : <BP1TULAP139.amoco.com> (this is the PSE instance) Click Next

8. Database System Common Parameters Database Charector Set: WE8DEC Database schema : SAPDAT DB Home : < E:\oracle\ora92>

9. User Information Name: preadm Password: Enter Password Confirm: Confirm Password

10. User Information Name:SAPservicePRE Password: Enter Password Confirm: Confirm Password

11. CD Browser CD Kernel : Enter the location of the Kernel CD

12. Monitor the Installation13. Filesystem Nodes

Check the filesystems/directories Click on Next

14. Extraction of SAP System Executables Select both Click on Next

15. Extraction of Oracle Software Select extract oracle client software Click on Next

16. SAP System Service Ports Enter/Check Port numbers (10) Click on Next

17. Monitor the Installation

3.4 <Basis> Prepare to restore PRE database to PFE3.4.1 Set the environment variables for PREadm

Go to Command Prompt Set Oracle_SID=PRE Set SAPDATA_HOME=F:\oracle\PRE Set DSMI_CONFIG= e:\usr\sap\PRE\SYS\exe\run\dsm_opt\.tsmsrv3.optNote: Do not close the command prompt session

3.4.2 Create the initPRE.utl fileUse the same command prompt session to execute the following command in order to create the initPRE.utl file on server BP1TULAP192 copy e:\oracle\ora92\database\initPFE.utl e:\oracle\ora92\database\initPRE.utl Edit the file e:\oracle\ora92\database\initPRE.utl and update the following:

o Change all occurrences of PFE to PREo Change MAX_SESSIONS to 8o Change MAX_BACK_SESSIONS to 4 and uncommento Change MAX_ARCH_SESSIONS to 4 and uncommento Change MAX_RESTORE_SESSIONS to 4 and uncommento Change SERVER to tsmsrv3o Change SESSIONS to 8o Change ADSMNODE to BP1TULAP131_PRE

of 12


o Change BRBACKUPMGTCLASS to A35_PRE3.4.3 Regenerate BKI file

Use the same command prompt session to execute the following command in order to create the initPRE.bki file: backint –p e:\oracle\ora92\database\initPRE.utl –f password Note: - Get the password from basis pool members

3.4.4 Obtain the backPRE.log file from tape Login to the database server as PREadm backint –p e:\oracle\ora92\database\initPRE.utl –f inquire Enter “# NULL e:\oracle\PRE\sapbackup\backPRE.log” backint –p e:\oracle\ora92\database\initPRE.utl –f restore Enter “<BackupID> e:\oracle\PRE\sapbackup\backPRE.log” Wait while the backPRE.log file is restored into the directory e:\oracle\PRE\sapbackup

3.4.5 Obtain the most recent backup’s logfile from tape View the file e:\oracle\PRE\sapbackup\backPRE.log to find the name of the most recent

complete PRE backup log. backint –p e:\oracle\ora92\database\initPRE.utl –f inquire Enter “#NULL e:\oracle\PRE\sapbackup\<backup_log>” The system will display a list of backups that hold this file, with the most recent one on top. Note

the backupID for the most recent copy of <backup_log>. backint –p e:\oracle\ora92\database\initPRE.utl –f restore Enter “<BackupID> e:\oracle\PRE\sapbackup\<backup_log>” Wait while the backup log is restored into the directory e:\oracle\PRE\sapbackup

3.4.6 Obtain the initPRE.ora parameter file from tape: backinit –p e:\oracle\ora92\database\initPRE.utl –f inquire Enter “# NULL \oracle\ora92\database\initPRE.ora” The system will display a list of backups that hold this file, with the most recent one on top. Note

the BackupID for the most recent copy of initPRE.ora. backint –p e:\oracle\ora92\database\initPRE.utl –f restore Enter “<BackupID> e:\oracle\ora92\database\initPRE.ora” Wait while the initPRE.ora file is restored into the directory e:\oracle\ora92\database

3.4.7 Obtain the initPRE.sap parameter file from tape: backinit –p e:\oracle\ora92\database\initPRE.utl –f inquire Enter “ # NULL e:\oracle\ora92\database\initPRE.sap” The system will display a list of backups that hold this file, with the most recent one on top. Note

the BackupID for the most recent copy of initPRE.sap. backint –p e:\oracle\ora92\database\initPRE.utl –f restore Enter “<BackupID> e:\oracle\ora92\database\initPRE.sap” Wait while the initPRE.sap file is restored into the directory e:\oracle\ora92\database

3.4.8 Obtain the initPRE.dba parameter file from tape: backinit –p e:\oracle\ora92\database\initPRE.utl –f inquire Enter “# NULL e:\oracle\ora92\database\initPRE.dba” The system will display a list of backups that hold this file, with the most recent one on top. Note

the BackupID for the most recent copy of initPRE.dba. backint –p e:\oracle\ora92\database\initPRE.utl –f restore Enter “<BackupID> e:\oracle\ora92\database\initPRE.dba” Wait while the initPRE.dba file is restored into the directory e:\oracle\ora92\database

3.5 <Basis> Restore PRE Files from Tape3.5.1 Restore datafiles

Use the same command prompt session to execute the following commands: brrestore -b <backuplog.anf> -d util_file -r e:\oracle\ora92\database\initPRE.utl -m full -l E This action should restore all datafiles and controlfiles to the system – when complete, verify the

restore completed successfully3.5.2 Restore Archive logs

Restore remaining PRE archive logs to PFE by logging in to the database servers

of 12


View the backup log file. Note the number next to “Current log sequence” at the top of the log file. This will be the first log needed, and the number you will use in the command below for <first log>.

Its necessary to restore as many logs as are on tape, so work with with the <lastlog> number accordingly in order to retrieve all logs that exist from tape.

brrestore -a <start_log#>-<end_log#> -c -d util_file -r e:\oracle\ora92\database\initPRE.utl The archivelogs will be restored into F:\oracle\PRE\oraarch

3.6 <Basis> Recover PRE3.6.1 Start Oracle Service for PRE

Start oracle service OracleServicePRE using computer management3.6.2 Roll forward database until last log

SQL> recover database using backup controlfile until cancel; Enter “auto” at the prompt to have the database automatically roll through all archivelogs in \

oracle\PRE\oraarch. Continue to roll forward until all available PRE archivelogs have been applied.

Note: If recover database fails with the following error:ORA-00283: recovery session canceled due to errors.ORA-01244: unnamed datafile(s) added to control file by media recoveryORA-01110: datafile <####>:’<full path of new datafile name>’Create a subdirectory for the data file under the appropriate sapdata filesystem.Then go back into server manager and enter the following command:

SQL> alter database create datafile '<full path of new datafile name>’ AS 'E:\oracle\PRE\920_64\database\UNNAMED0<####>’;Where <full path of new datafile name> and <####> here is the same as <full path of new datafile name> and <####> in the error message above.

From this point you should be able to resume rolling forward by reissuing the recover database command above.

3.7 <Basis> Open PRE databaseIn this step the database logs are reset; this is a key activity; if it fails the database restore and log recovery need to be done over again. Login to the database server as PREadm sqlplus /nolog SQL> connect / as sysdba SQL> alter database open resetlogs;

The result should be a message saying, “Database opened”Verify the database opened successfullyPlease check the oracle alert log for more info.

SQL> shutdown SQL> startup Ensure that normal shutdown and startup work correctly.

3.8 <Basis & IBM Tulsa> Update /usr/sap/trans The /usr/sap/trans filesystem may have to be recreated and restored depending on the nature of the disaster. (Currently the PSE system, BP1TULAP139, holds the /usr/sap/trans filesystem.)

3.8.1 <Basis> Create /usr/sap/trans directory Create the directory /usr/sap/trans

3.8.2 <IBM Tulsa> Restore /usr/sap/transIt may be necessaryRestore the contents of the last /usr/sap/trans backup into the new directory: Login to the new server housing /usr/sap/trans as admin dsmc restore –server=tsmsrv3 –virtualnode=tdcsapdn –subdir=yes /usr/sap/trans/*

3.9 <Basis> Configure Oracle Files and Start Listener Login to the database server as PREadm

of 12


Open tnslistener file Change all occurrences of PFE to PRE in this file Ensure that the listener is configured to run on port 1521

3.10 <Basis & IBM Tulsa> Initial Start & SAP Configuration3.10.1 <IBM Tulsa> Update DNS

Update any DNS entries pointing to BP1TULAP181 & BP1TULAP136 to now point to BP1TULAP192 & BPTULAP113 respectively.

Currently known DNS entry is: SAPPRE pointing to BP1TULAP1813.10.2 <Basis> Request SAP license

An SAP license must be requested to run PRE on BP1TULAP113 instead of BP1TULAP181 Obtain this new license from SAP via the OSS. To request an emergency temporary license, call

1-800-677-7271.3.10.3 <Basis> Install New SAP License

Login to the central instance as PREadm C:\> saplicense –show C:\> saplicense –deleteInput the information requested to delete the old license referencing PFE C:\> saplicense –installInput the information requested for the new license

3.10.4 <Basis> Restore SAP profiles from tape backinit –p e:\oracle\ora92\database\initPRE.utl –f inquire Enter “# NULL <full name and path of profile>” The system will display a list of backups that hold this file, with the most recent one on top. Note

the BackupID for the most recent copy of the profile you are looking for. backint –p e:\oracle\ora92\database\initPRE.utl –f restore Enter “<BackupID> <full name and path of profile>” Wait while the profile file is restored Repeat this process for all profiles needed (PRE_DVEBMGS10_<hostname>, DEFAULT,

START_DVEBMGS10_<hostname>, etc.)3.10.5 <Basis> Update SAP profiles as needed

Compare the SAP profiles installed as part of the installation process to the profiles just restored from tape

Update the profiles accordingly3.10.6 <Basis> Remove Batch and Spool Processes from the system

On the central instance, BP1TULAP113, edit the Central Instance SAP profiles commenting out the lines, which reference ‘rdisp/wp_no_btc’ and ‘rdisp/wp_no_spo’.

The SAP profiles can be found on the central instance server in e:\sapmnt\PRE\profile3.10.7 <Basis> Start PRE Central Instance

Logon to the central instance BP1TULAP113 as user PREadm Ensure that /usr/sap/trans is mounted to the system Start SAP using SAP MMC

3.10.8 <Basis> Lock All users Using transaction su01, perform a mass change to lock users.

3.10.9 <Basis> Import instance profiles Call transaction RZ10. Import the instance profiles by selecting Utilities –> Import profiles –> of active servers from the

top menu.3.10.10 <Basis> Operation Modes

Go to transaction rz04 Change the existing Operation Modes to reflect the hostnames of the new servers.

3.10.11 <Basis> Execute the initial SAP consistency check Call transaction SICK. The only error reported should be one stating that there are no batch processes currently

defined. This error is acceptable since there shouldn’t be any batch process online yet.3.10.12 <Basis> Check Partner profile in PRE

of 12


Check RFC destinations with in SM59 Go to WE20 Select partner 'PRECLNT100' Check the RFC destination for port to 'PRECLNT100' Check all the outbound and inbound parameters Activate the partner profile (if required) Go to WE20 Select partner ‘BP_TP5_100’ Check the RFC destination for port to A000001 Check all the outbound and inbound parameters Activate the partner profile (if required)

3.10.13 <Basis> RFC Server Groups Call transaction RZ12 Recreate all PRE Server groups related to old hostnames with new hostnames Double click on bp1tulap181_PRE_30 Enter ‘parallel_generators’ as server group Click on Save

3.11 <Basis> Final Start of PRE with BatchIMPORTANT: BEFORE DOING THIS STEP COORDINATE WITH THE BUSINESS / CASSIS Wave 1 PROJECT SUPPORT TEAM. These teams may ask you to do any or all of the following steps in any order:

3.11.1 To Activate the batch and spool processes Shutdown the Central Instance. Modify the Central Instance SAP profile, un-commenting the entries, which reference

‘rdisp/wp_no_btc’ and ‘rdisp/wp_no_spo’.3.11.2 Resume Batch Scheduling

The Escalation Process Team will be assembled to identify all tasks surrounding the outage. The following tasks will be performed after an outage to catch up processing: Perform reconfiguration tasks for any interfaces that need it. Work with the Functional Experts to analyze the queue of jobs that are waiting to be processed Check for sequence/dependency issues. Reschedule dependent jobs to run after predecessor. Determine business critical jobs that must run first. Coordinate application server start-up with the Basis Team If the outage is extensive (>12 hours), the jobs that are allowed to process may have to be

limited, based on job class. Basis will modify the application server settings to allow only class A jobs to process first, then class B and C.

For a shorter outage, Basis may be asked to bring up all application servers with full batch processing.

Once the application servers are up, with full batch processing, and all jobs have been reviewed, the Functional Experts manually catch up the batch schedule by running jobs that were missed during the outage.

3.11.3 Start PREStart PRE Central Instance Logon to the central instance, BP1TULAP113 as user PREadm Ensure that /usr/sap/trans is mounted to the system C:/> startsapTo modify batch processes to run Class A jobs only: Go to transaction rz04 Double click on the current op mode running that you want to modify Double click on the current operation mode listed under the app instance you want to modify Change the batch jobs so that the numbers of class A batch jobs is equal to the total number of

batch jobs Save your changes Go to transaction rz03

of 12


Double click on the app instance you changed so its highlighted From the menu, choose Control –> Switch Op Mode –> of selected servers

3.11.4 Setup Login Groups Go to transaction SMLG. Change the existing login groups to reflect the new servers.

3.12 <Basis> Final Steps3.12.1 Resume Operations procedures

Reconfigure TSM Server for Backups: Login to BP1TULAP192 E:\> cd /oracle/PRE/920_64/database Update the file initPRE.utl and change ADSMNODE to BP1TULAP113_PRE Start Online Backup

Use DB13Schedule immediate online backup

Schedule Archive log backups and Online BackupsUse DB13Schedule Archive log backups – HourlySchedule Online Backups – Every Week

3.12.2 Configure STMS Login to PSE client 000 as user sap* Go to STMS Click on systems Select PFE Click on Delete Distribute and update the configuration

Login to PRE client 0000 as user sap* Go to STMS You will get a popup window. Confirm that transport domain is DOMAIN_PSE and domain controller is PSE Now click on Save

Login to GRM client 000 as user sap* Go to STMS Click on systems Select PRE Click on Approve Distribute and update the configuration

Click on Transport routes Click on Change Select Standard configuration-> Three system in Group from the menu Enter the following, Development system: PSE Quality assurance: PFE Production system: PRE Click on Save Distribute and update the configuration

of 12