the use of hds’s and emc’s dasd copy services at gary spencer worldspan
TRANSCRIPT
The Use of HDS’s and EMC’s DASD Copy Services at
Gary Spencer
Worldspan
The Use of HDS’s and EMC’s DASD Copy Services at Worldspan
Terminology•Target volume•Source volume•Native test system•Periodic refreshed test system•Daily test system•Read/write VM test system
The Use of HDS’s and EMC’s DASD Copy Services at Worldspan
External Control Unit Copy•“Remote copy”
•Runs external of host •Full volume copy•“Sync’ed” up•Synchronous (semi-sync, adaptive copy) •Suspend/split•Point in time copy•Incremental refresh (copy)•Restore capabilities •Higher Impact on Host
Source Ctl Unit
100
To Hos
t
Fiber connection
Target Ctl Unit
101
May
be T
o Hos
t # 2
xxx
yyy
The Use of HDS’s and EMC’s DASD Copy Services at Worldspan
Internal Control Unit Copy
•“Local Copy”•Runs external of host •Very fast – low host impact•Full volume copy•“sync’ed” up•Suspend/split•Point in time copy•Incremental refresh (copy)•Restore capabilities
Ctl Unit
100
101
To Hos
t
Target volume are “generally” not addressable to the same host as the Source volumes. (IOCP’s are generally used at Worldspan to control access)
May
be T
o Hos
t # 2
xxx
yyy
The Use of HDS’s and EMC’s DASD Copy Services at Worldspan
Vendor Specific Comments
Product Names
•TrueCopy - (HDS’s remote copy)•Symmetrix Remote Data Facility - SRDF (EMC’s remote copy)•Shadow Image - (HDS’s local copy)•TimeFinder - (EMC’s local copy)
Target/Source pairing
• TrueCopy and Shadow Image may be defined in the host or by the CE in the control unit
•SRDF source/target pairs may be defined in the host and must be defined by the CE in the control units BIN file.
•TimeFinder pairs are call Standard volumes/BCVs (Business Continuance Volume)•Must be defined in the control unit as “Standard” volumes or as “BCV’s”•Source/Target (standard/BCV) pairs are defined in user controlled host tables•Copies may be from any “standard” volume to any “BCV”.
The Use of HDS’s and EMC’s DASD Copy Services at Worldspan
Worldspan’s Three Production TPF systems
•OSS - Delta Airline’s flight control and flight operations system
•Deltamatic (RES) - Delta Airline’s reservation system
•GDS - Worldspan’s agency reservation system Northwest Airline’s reservation system
Use of HDS’s Copy Services on Worldspan’s GDS
Initial Goal – speed up test system buildFirst use of copy services to build test systemsFirst use of “local” copy
Use of HDS’s Copy Services on Worldspan’s GDS The Production GDS System
•GDS is a 8 way loosely couple TPF system•GDS is Worldspan’s agency reservation system •GDS is Northwest Airline’s reservation system
Statistics•GDS has 640 prime mods (1280 active mods)•Each physical control unit has 45 or 46 active modules•The I/O rate is approximately 250,000 + I/O’s per second
Fir
eW
all
28 HDS 7700E’s3 LCU’s per CU HDS
7700E HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E HDS
7700E
HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E
HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E
HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E
Use of HDS’s Copy Services on Worldspan’s GDSPrior GDS Test Environment
“Large Test System”
Prior GDS test environment- ltst3 VM Systems
Tape Restore
Large Test (LTST)4 7700E’s
HDS 7700E HDS
7700E HDS 7700E HDS
7700E
Any GDS production processor
Large Test System runs read/write to VM during the week – this read/write system is a customer system.Down Time is considered an impact.System is refreshed each Sunday from Saturday capture tapes.Native testing was restricted to weekends before the refresh (some exceptions allowed)The LTST DASD has connectivity to each of the GDS production processors.
Use of HDS’s Copy Services on Worldspan’s GDSCurrent GDS Test Environment
Large Test
HDS LTST 7700E - 1
•4 physical CU’s•2 copies of the production database•TrueCopy’s target from the production system rotates each week.•Shadow copy the new TrueCopy’ed test system to other address range•Unique VSN for each address range
HDS 7700E
Addr range xx00-xx3f
Addr range xx40-xx7f
Use of HDS’s Copy Services on Worldspan’s GDSCurrent GDS Test Environment
Large Test
Current GDS test environment - ltst
3 VM Systems
4 7700E’s 1 set of shadow volumes
HDS 7700E HDS
7700E HDS 7700E HDS
7700E
Any GDS production processorOr VM LPAR
Fir
e
Wall
HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E
HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E
14 7700E’s1 set of shadow volumes
Shadow volumes
Rotating TrueCopy Targets
Low addresses
High addresses
Prod volumes
Native or VM read/write testing use the non-ltst volumes
Step one – shadow productionStep two – TrueCopy to LTSTStep three – build LTSTStep four – IPL new databaseStep five – shadow LTST
Use of HDS’s Copy Services on Worldspan’s GDSPrior GDS Disaster Recovery
Prior GDS GDS Disaster Recovery
Tape Restore
Use of HDS’s Copy Services on Worldspan’s GDSCurrent GDS Disaster Recovery
Current GDS Disaster Recovery
Fir
eW
all
28 HDS 7700E’s3 LCU’s per CU
HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E
HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E
HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E
HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E HDS
7700E HDS 7700E
•A shadow image copy of the GDS database each night.•System may be restored from the shadow’ed volumes.•Groundwork for a no tape capture
shadow volume
Productionvolume
Use of HDS’s Copy Services on Worldspan’s GDS
TPF Software Use
TPF software use
•First use of HDS’s TPF software for TrueCopy and Shadow Image•Originally, Shadow Image and TrueCopy activities were done manually by CE
•Production control improvements•TPF Coverage staff controls the activity•Automation is able to run the “copy” procedures.•The TPF software eliminated the need for logging tapes during the database split
•TPF software now controls all TrueCopy and Shadow Image activities for test system builds and DR support
•Software was new
Use of HDS’s Copy Services on Worldspan’s GDSBenefits to GDS
GDS payoffs
1. Test System benefitsa. Improved LTST down/build time (18 plus hours to less than 2 hours) b. Removing tape restore from LTST build eliminated tape prep/sort/errors time.c. Better and More Native Testing capabilities
i. No down time for native testingii. Native testing at any time – no impact on ltst databaseiii. Native test time setup takes about 30 minutes (using some of VM’s LPAR’s )
d. Read/write ltst system under VMe. Quick copies of small test systems (8 mod test system)
2. Disaster Recovery benefits.a. Laid groundwork to eliminate tape captureb. Production database can be restored (quickly) from the shadow image copy.
3. New DASD is often formatted using the control unit’s copy services.
4. Proved TrueCopy and Shadow Image technology for TPF use at Worldspan5. Proved TrueCopy and Shadow Image TPF software controls for use at Worldspan
Use of HDS’s Copy Services on Worldspan’s GDS
GDS Current Development
Current development
• Working with HDS to use shadow image to split the production database on a
nightly basis • Working with HDS and with Worldspan’s coverage, production control staffs
to improve automation and production control issues• Training Worldspan Coverage Staff
Possible Future GDS Development
• Another set of shadow volumes on the large test system (allow to create a daily test system)
• More shadow volumes on the production database (allow multiple backup copies of the production database)
• Continue to work towards a capture without tapes.
Use of EMC’s Copy Services on Worldspan’s Deltamatic System
•Most ambitious goals•Most work done initially with TPF software•EMC provided an onsite consultant•First Use of Timefinder
•SRDF was already in use on OSS
Use of EMC’s Copy Services on Worldspan’s Deltamatic System The Production Deltamatic (RES) System
Fir
eW
all
•Deltamatic is a 6 way loosely coupled TPF system•Deltamatic is Delta Airline’s reservation system
EMC 8430
Statistics•Deltamatic 592 prime mods (1184 active mods)•Each EMC 8430 has 74 active modules•Each EMC 5830 has 53 or 54 active mods •The I/O rate is approximately 115,000 I/O’s per second
EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430EMC
8430
EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830
11 EMC 5830’s2 LCU per CU
8 EMC 8430’s2 LCU per CU
Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemPrior Deltamatic Test Environment
Prior Deltamatic test environment2 VM Systems
PMR
Tape Restore
•Native testing restricted to quarterly DR test weekends•No daily test system•Intermediate step to replace PMR with 5430’s and manual SRDF•Brought in the 8730 to upgrade the testing facilities
Deltamatic Test System (RTS)
Fir
e
Wall
EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830
11 EMC 5830’s2 LCU per CU
EMC 8730
SRDF LINK
EMC 5430
Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemThe EMC 8730 Control Unit for Three Test Systems
EMC 8730 - 1
•10 LCU’s•3 copies of the production database•Each LCU has three 64 address ranges•Unique VSN for each address range•11 production control units (22 LCUs) mapped to 10 test system LCUs.
EMC 8730
Addr range xx00-xx3f
Addr range xx40-xx7f
Addr range xx80-xxbf
LCU 1
Addr range xx00-xx3f
Addr range xx40-xx7f
Addr range xx80-xxbf
LCU 10
…
Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemThe EMC 8730 Control Unit for Three Test Systems
EMC 8730 - 2
•First address range in each LCU•The daily test system, a base for VM VPAR systems •Target volumes for SRDF from the production system (refreshed nightly)• TimeFinder’s standard volumes
•Second address range in each LCU•The periodic test system, a base for VM VPAR systems •These devices are mirrored•TimeFinder’s BCV group 1
•Third address range in each LCU•The native test system or a read/write test system for VM •TimeFinder’s BCV group 2
Addr range xx00-xx3fTimeFinder std volumes
Addr range xx40-xx7fTimeFinder BCV group 1
Addr range xx80-xxbfTimeFinder BCV group 2
8730’s LCU x
Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemCurrent Deltamatic Test Environment
Current Deltamatic test environment
2 VM Systems
SRDF linksStd to std
Fir
e
Wall
EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830
11 EMC 5830’s2 LCU per CU1 std, 2 bcv’s each volume
EMC 8730
2 Deltamatic production processors
standard
BCV 1
BCV 2
Loosely coupledNative testing
•VM VPARS Daily test system•VM VPARS Periodic test system
•No more tape restore •VM read/write test system
Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemPrior Deltamatic Disaster Recovery
Prior Deltamatic Disaster Recovery
Fir
e
Wall
EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430
8 EMC 8430’s2 LCU per CU
…
EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430
8 EMC 8430’s2 LCU per CU
SRDF linksContinuous, semisync
Wor
ldsp
an D
ata
Cen
ter
DR
Site •Break SRDF links for …
•DASD for quarterly DR test exercise•Quarterly native test DASD•Selective database recovery from
capture or logging tapes
Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemCurrent Deltamatic Disaster Recovery
Current Deltamatic Disaster Recovery
ProductionEMC
5830 and 8430
Standard volumes(production active volumes)
BCV 1 Copy to from standard
BCV 2 Copy to from standard
Alternate TimeFinder copy from standard volumes to BCVs
Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemCurrent Deltamatic Disaster Recovery
Current Deltamatic Disaster Recovery
Fir
e
Wall
EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430
8 EMC 8430’s2 LCU per CU
EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430EMC
8430EMC 8430
8 EMC 8430’s2 LCU per CU
SRDF linksContinuous, semisync
Wor
ldsp
an D
ata
Cen
ter
DR
Site
•Control DR SRDF stop/start with TPF software•In production, we can “capture” the database rotating between two BCV’s.•At the DR site, we can “capture” the database rotating between two BCV’s.•Selective database recovery from the daily test system•Production system may be restored from the BCV’s
EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830
EMC 5830EMC
5830EMC 5830EMC
5830EMC 5830EMC
5830
11 EMC 5830’s2 LCU per CU
Each standard has 2 BCV’sEach standard has 2 BCV’s
Each standard has 2 BCV’s
Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemTPF Software Use
TPF software use
•First use of EMC’s TimeFinder software•EMC’s TPF SRDF software had been used on OSS•Similar advantages as the GDS
•TPF Coverage Staff controls the activity, software controls are automated
•No tapes (including logging tapes) required for test system refresh
•The TPF software controls all SRDF and TimeFinder activities for test system builds and DR activities
•Working to use remote TimeFinder for DR copies.
•TimeFinder must be started with host software (not a routine task for a CE)
•The software can be used to restore the production database from the BCVs..
Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemBenefits to Deltamatic
Deltamatic payoffs
1. Test System benefitsa. Native Testing capabilities on request (no longer must wait for a quarterly DR test)b. Daily Test System
i. Improved problem analysis from 6 wk system c. A read/write VM system for high write testing d. Periodic Test System build much easier, faster (no tapes)
2. Disaster Recovery benefits.a. Coverage controls DR SRDF activity with TPF softwareb. Laid groundwork to eliminate tape capturec. Have multiple days of copies of the database on DASDd. Daily Test System
i. Selective database restoreii. Eliminate the need for continual logging tapes (although not done yet)
3. Proved TimeFinder and remote TimeFinder technology for TPF use at Worldspan’s 4. Proved EMC’s TPF TimeFinder software for use at Worldspan’s
Use of EMC’s Copy Services on Worldspan’s Deltamatic System
Deltamatic Current Development
Current and future development
•Completing the implementation in using BCV’s for database capture/restore•Working with EMC to complete implementation of remote TimeFinder use•Working with EMC to use TimeFinder in the production system•Working with EMC to improve automation and production control issues•Training Worldspan Coverage Staff
• Work towards a no tape capture solution• More BCV’s for more “back” days of capture
Possible Future Deltamatic Development
Use of EMC’s Copy Services on Worldspan’s OSS
•Limited Goals due to older Technology used for this system •EMC 5230’s – no TimeFinder •Primary goal was to set up a native test environment
•Initial use of EMC’s TPF software
•Work similar to the Deltamatic work
Use of EMC’s Copy Services on Worldspan’s OSS The Production OSS System
Spl
it
Site
(Disaster Recovery Location)(Worldspan Data Center)
•OSS is a Uni-Processor TPF system•OSS is Delta Airline’s flight control and flight operations system
EMC 5230
EMC 5230
EMC 5230
EMC 5230
Statistics•OSS has 72 prime mods (144 active mods)•Each physical control unit has 36 active modules•The I/O rate is approximately 3500 I/O’s per second
4 EMC 52300’s2 LCU per CU
Use of EMC’s Copy Services on Worldspan’s OSSBenefits to OSS
OSS payoffs
Similar to the Deltamatic benefits.
By adding a new control unit for testing – and using EMC’s copy services -
1. Test system has native testing and a daily test system
2. Disaster Recovery gained from the daily test system and easier setup for DR testing. Also, the groundwork was started to help eliminate tape capture
3. Proved SRDF technology and TPF software controls for TPF use at Worldpan.
With new technology - OSS could make use of TimeFinder in similar ways as Deltamatic.
Use of EMC’s Copy Services on Worldspan’s OSSRecap of the OSS work
Spl
it Site
(Disaster R
ecovery Location)
(Wor
ldsp
an D
atac
ente
r)
recap of oss work
Production OSS Processor
EMC 5230
EMC 5230
EMC 5230
EMC 5230
EMC 5230
DR OSS Processor
RES Processor
PMR
EMC 5230
SRDF links
Tape Restore
2 VM Systems
Daily test systemNative test system
SRDF links
Manual process
AMOD UP
•DR duplicate DASD•DASD for DR test exercise•Quarterly native test DASD
The Use of HDS’s and EMC’s DASD Copy Services at Worldspan
Implementation Experiences
Common Experiences between Vendors or between TPF Systems
Production I/O rate impacted copy servicesMost copy service problems were found during production useStalled modsHad to switch (or will switch) to use local copy only on production. Then remote copy from that local production copy.
The TPF software commands were done manually until the concept was proven. At that time the software commands to control the copy activity were automated Many Automation/Production Control issues
Coverage education. Specifically awareness that this activity runs outside of TPF ’s control. CE’s education.
Outstanding Vendor and Vendor’s programmer support. Code was high quality - errors found were generally due to configuration issues. Vendor’s would provide off hour telephone or onsite support.
Building source/target tables required vendor help.
OCO code delayed testing, especially during production testing (examples : waiting for programmers to resolve problems, waiting for new code).
The Use of HDS’s and EMC’s DASD Copy Services at Worldspan
Implementation Experiences
Experiences with the Delta Systems and EMC
Need for rebuilding BIN files during implementation and testingEMC provided a full time onsite consultant for the TPF software implementation and to help set up the use of TimeFinder
Experiences with the GDS and HDS
TPF software was new
The Use of HDS’s and EMC’s DASD Copy Services at Worldspan
What Worldspan Gained
•Test Systems•Easier access to native testing on all three systems•A “daily” test system on both of the Delta Airline TPF systems•Faster “Periodic” Test system build for GDS and Deltamatic•Test systems built using no tapes (reduced tape activities)•Less test system down time (“or” test systems are more available)•Daily test systems, native test systems are increasing the quality of testing
•Disaster Recovery•Easier (and faster) to set up a DR test for Delta Airlines.•Quick access to a 24 hour old database for selective database restores•All three systems have a DASD copy of the database to restore from• Started Groundwork for a no tape capture
•More ?•As the programming groups become experienced using these test environments, we have seen them exploit this tool and continue to come up with new ways to use these systems. Today on GDS we are seeing extensive recoup/capture testing.