brad mccusker software concepts international, llc phone...

Copyright 2017

SCI, LLC, Nashua, NH USA

Software Concepts International, LLC.World Class Managed Services For OpenVMS

Brad McCuskerSoftware Concepts International, LLC57 Technology Way, Suite 2W1Nashua, NH 03060, USA

Phone: 603-879-9022

e-mail: [email protected]

www.sciinc.com

Proven results by applying the

best practices in Computer

Systems Management

Copyright 2017



� Financial Services – can we do it?

� Latest & Greatest as long as nothing changes

� VAX to Alpha with 90% of the pieces

Copyright 2017



Managing OpenVMS systems and databases requiring the highest

levels of performance and availability – worldwide

Copyright 2017



4

Copyright 2017



5

� Replace aging DS-25 & MSA

� Virtualize the platform

Copyright 2017



� DS25 with 1GHz processor

� Concern over emulator performance

� Could we satisfy the extended periods of high CPU?

� In 2010 – told customer low confidence of success

6

050

100150200250

9:4

1:2

4 …

10

:16

:33

…

10

:51

:43

…

11

:26

:52

…

12

:02

:02

…

12

:37

:11

…

1:1

2:2

1 P

M

1:4

7:3

0 P

M

2:2

2:4

0 …

2:5

7:4

9 …

3:3

2:5

9 …

4:0

8:0

8 …

4:4

3:1

7 P

M

5:1

8:2

7 …

5:5

3:3

6 …

6:2

8:4

6 …

7:0

3:5

5 P

M

7:3

9:0

5 …

8:1

4:1

4 …

8:4

9:2

4 …

9:2

4:3

3 …

9:5

9:4

3 …

10

:34

:52

…

11

:10

:02

…

11

:45

:11

…

12

:20

:21

…

12

:55

:30

…

1:3

0:4

0 …

2:0

5:4

9 …

2:4

0:5

9 …

3:1

6:0

8 …

3:5

1:1

8 …

4:2

6:2

7 …

5:0

1:3

6 …

5:3

6:4

6 …

6:1

1:5

5 …

6:4

7:0

5 …

7:2

2:1

4 …

7:5

7:2

4 …

8:3

2:3

3 …

9:0

7:4

3 …

% CPU Idle Time

%Idle

Copyright 2017



� Existing� DS-25, 2P (EV68 1.0GHz), 10GB

� OpenVMS 7.3-2

� MSA 1500, 3 volumes, ~600GB

� Proposed� Dell PowerEdge R630

� 2 - E5-2637 v3 3.5GHz,4C Processors, 128GB

� VMware ESXi 6.0

� Windows 2012

� Digi AnywhereUSB (for CHARON key)

7

Copyright 2017



� Boot emulator with temporary VMS

� Network transfer of savesets

� Shut down Alpha

� I hated to bring this one down:

� Reconfigure, reboot emulator

� Overall, uneventful migration

8

AXP>show system /noproc

OpenVMS V7.3-2 on node XXXX 16-MAY-2016 12:10:13.08 Uptime 1317 15:15:03

AXP>

Copyright 2017



� Winsock 10035 error when trying to connect from clients

� Google, etc. indicated firewall problem

� Network team disabled a firewall somewhere

� Problem solved

� BZZT!

� Many days later, same problem, firewall still disabled

� Lots of wasted time chasing phantom firewall issues.

� Ultimately resolved by executing @CNXSVR again

9

Copyright 2017



� CPU:

� Our experience with ProLiants is CHARON/VMWare should be around 2277 VUPs

� Dell: 94% as fast

� Followed DELL guidelines for high performance, simply couldn’t make it faster

10

CHARON: Approximate VUPs Rating : 2149.3 (min: 2121.0 max: 2167.0)

DS25: Approximate VUPs Rating : 2542.8 (min: 2538.0 max: 2547.0)

Copyright 2017



� Initial results: batch jobs runabout 25% slower

� 50-70% CPU, 1500-2000 DIO

� Tuning:SET RMS/BLO=127/BUF=8/SEQ and

SET RMS/BUF=128/IND

SET RMS_SEQFILE_WBH =1

SET RMS_DEFAULT/QUERY_LOCK=DISABLE

�Batch job performance went to 90%

11

Copyright 2017



� Alpha: MSA 1500

� 15K SAS drives

� CHARON configured with “raw SCSI” devices

� VMware .vmdk actually

� Started with NetApp SAN

� 1 GB Ethernet, 15K SAS drives

� Next: DellSCv2020 ISCSI 10GbE Dual Controller

� 1GB switches, etc. 10K SAS drives

� NetApp was temporary use only

� Separate project replacing NetApp with Dell

12

Copyright 2017



� I/O with NetApp: Faster than MSA

� I/O with Dell: Slower than the MSA

� Customer was concerned

� Proposed using Windows I/O caching

� Risk: I/Os in Windows cache could be lost

� Customer willing to accept that risk

13

Copyright 2017



14

0

2000

4000

6000

8000

10000

12000

14000

16000

1 2 4 8 16 32 64

Xfer/sec

Q Depth

256 block Random Writes

Container with windows cache DS-25 & MSA VMDK Raw SCSI

Copyright 2017



� Switch to Windows cache resulted in applications running at 95% of the Alpha

� Acceptable results, customer happy!

15

Copyright 2017



� Current procedure:

� Interactively run a procedure

� Walk into the computer room and insert/swap tapes.

� Every day!

� Most tapes kept onsite – Friday tapes sent offsite.

� New process:

� Backup to virtual tape on VMWare guest

� Virtual tape is a windows container file

� Virtual machine gets backed up to local repository

� Copy of the virtual machine is sent to the cloud

16

Copyright 2017



� Fully automated, VMS Batch

� No more physical tapes

17

Copyright 2017



18

Copyright 2017



� Alpha � Integrity

� EVA � 3PAR

� Custom code migration

� Servers hosted/managed by 3rd party with limited modern VMS knowledge

� Move new systems in-house

19

Copyright 2017



� GS1280, 2 partitions (12P/4P)

� EVA5000, dual controller, Business Backup� 168 36GB 18K drives, RAID 0+1

� BASIC, C, COBOL, Datatrieve, DECForms, CMS SNA/LU6.2

� Oracle Rdb/CDD

� IBM MQ Series

� Xentis, AIG UDMS (report generation)

� CA PA(VPA), DC

� Not available IA64: Raxco Control, FaxSr, Eigen Utilities, SuperSqz

20

Copyright 2017



� 2x HP Integrity BL870 servers� 4 x I2 9350 (1.73GHZ/4C), 64GB

� C7000 enclosure

� 3PAR 7200C – 2 nodes

� 24 x 920GB SSD

� Test/Dev: RX2800, 2P I2 9340 (1.66GHz)/4C

� Software: Same as Alpha, except:� NDS/Compufax, Peek/Spy, xxxx?

21

Copyright 2017



� 95% BASIC, rest is C, COBOL, Macro

� TOLAS Run time libraries (owned sources)

� RMS files for data

22

Copyright 2017



� Provided loaner IA64 server for intial porting POC

� Configured dev/test in NJ data center� Difficulties with ILO address

� Migrated from loaner� Storage in NJ????

� Customer moved dev/test and 3 PAR to CT data center

� Assist with planning for enclosure switching, 3PAR configuration

� Configured C7000 and blades in CT data center

� HP configured 3PAR� HP ‘expert’ really wasn’t

23

Copyright 2017



� Old system, from VMS, � Imitated an EVA based snapshot of disk to be backed

up

� Present snapshot via SSSU CLI to VMS

� VMS Backup to tape used against privately mounted snaps

� New system,� Plan was to Use 3 PAR Virtual Copy from VMS using

the Python-3parclient toolset

� Didn’t work out

� Used SSH to 3PAR to execute 3PAR commands. 3PAR snaps, then used VSM backup on private volumes

24

Copyright 2017



� Significant work to emulate current installation while taking advantage new technology

� Different customer team members lacked “big picture” knowledge or view

� SCI provided big picture oversight

� Systems reside in a hosting company’s Data Center

� Some of the network was controlled by the hosting

company

� Hosting personnel’s first HP blade enclosure

25

Copyright 2017



� Customer handled all testing

� Didn’t do enough

� “We tested what we thought was important”

� Didn’t test things that were difficult to replicate

� FTP, SFTP, FAX, SNA

� WebsphereMQ

26

Copyright 2017



� Delayed due to testing issues.

� SCI engaged to help troubleshoot

� Ultimately successful, couple week late

� SCI retained to provide Managed Services.

27

Copyright © 2016


Software Concepts International, LLC.29 September 2017

Copyright © 2016



� Financial sector customer wishes to upgrade critical production environment to current supported VSI OpenVMS and Oracle Rdb releases

� SCI engaged to perform production system and database upgrades and provide post-upgrade analysis and support

� Engineer forced to do international travel

29

Copyright © 2016



� Audit 10 months ago identified unsupported database and operating system in production

� Planning for upgrades failed to achieve required priority

� Production upgrade rush schedule� No test/development upgrade (more about this later)

� Limited testing

� No application changes – only VMS and Rdb and related products / options

30

Copyright © 2016



� Upgraded production system Sunday morning early� Branches open at around 10:00

� Upgrades and installs went well

� Database converts failed due to logical name complexities (process, group and system) from different upgrade account (SYSTEM vs application)� Quick restore of database

� ‘Smoke tests’ for an hour before full production

32

Copyright © 2016



� Average can further be significantly improved with database tuning

34

Node: ACCPQ (1/1/16) Oracle Rdb V7.3-210 Perf. Monitor 25-JUL-2017 09:49:42.39

Rate: 3.00 Seconds Transaction Duration (Total) Elapsed: 08:52:23.52

Page: 1 of 1 DISK$SBS:[STFEXP.MAE]SBS.RDB;2 Mode: Online

--------------------------------------------------------------------------------

Total transaction count: 58829

Seconds Tx.Count: % #Complete: % #Incomplete: %

0-< 1: 57675 98% 57675 98% 1154 2% <-avg=0.167410 95%=0.20

1-< 2: 462 0% 58137 98% 692 2%

2-< 3: 235 0% 58372 99% 457 1%

3-< 4: 194 0% 58566 99% 263 1%

4-< 5: 79 0% 58645 99% 184 1%

5-< 6: 25 0% 58670 99% 159 1%

6-< 7: 8 0% 58678 99% 151 1%

7-< 8: 15 0% 58693 99% 136 1%

8-< 9: 16 0% 58709 99% 120 1%

9-<10: 1 0% 58710 99% 119 1%

10+++: 119 0% 58829 100% 0 0%

Copyright © 2016


Software Concepts International, LLC.29 September 2017 35

OLTP STF SBS

Database

AIJ

JCC

LogMinerLoaderRdb Continuous

LogMiner

• Near real time CDC + ETL• OLTP data available for read

access shortly after commit• Change data collected from

live AIJ files – no database access

• Zero / minimal OLTP impact• Offload read-only access to

SQL Server• Via JCC LogMinerLoader

“Mirror” SQL

Server DatabaseAIJ

AIJAIJ

Copyright © 2016



� Required upgrade to Rdb 73 and VMS 842L2

� Mandatory in order to enable restore of production

databases to test/development environment

� Optional migration to CharonAXP-AXP on VMware to improve performance and reliability

36

Copyright © 2016



� More processing units

� Increase system headroom

� Permit additional jobs to run in parallel

� Does not improve single-job performance; increases

system capacity

� Current system 2 emulated processors

� Charon ES40 Emulator supports 4 processors

� Requires additional CPU licenses

37

Copyright © 2016



� [TCPIP$DHCP]TCPIP$DHCP_RUN.LOG was the busiest file on the system disk

� Full of error messages

� DHCP server is not required nor desired on this system

� The DHCP Server has been disabled

38

Copyright 2017



� Located in Nashua, NH (USA)

� 30 years in business supporting OpenVMS!

� International reputation as leading provider of

� Managed services for OpenVMS & databases

� OpenVMS performance and consulting services

� Proven global track record

� Actively managing 100s of systems and databases

� At many sites 24x365 since 1995

Copyright 2017



� System and database performance consulting

� VAX/Alpha emulation – CHARON reseller

� Migration consulting

� Specializing in minimal downtime migrations

� Oracle’s worldwide provider of CODASYL DBMS training

41

brad mccusker software concepts international, llc phone...

Documents