sysplex in a nutshell

®

IBM Software Group – Enterprise Networking Solutions

z/OS® Communications Server

z/OS Communications Server Development, Raleigh, North Carolina

Sysplex In a Nutshell

Andrew Arrowood - [email protected]


Interface 21:

IP Addr 2234::2

Interface 22:

IP Addr 2235::2

Background – Virtual IP Address

Application access independent of network topology

– VIPA identifies the server application, not the physical interface

– Physical interface IP addresses can change without affecting end-user

– No changes needed in DNS or hosts file configuration

Interface resilience

– Communication with a server host is unaffected as long as a physical interface is active

– Example: As long as interface 21 or 22 is active, the connection to the CICS application stays active

TCPIP1 CICS

201.81.1.0

PORT 5000

Advertise: VIPA 201.81.1.0


DYNAMICXCF Interfaces

TCP/IP

Stack-1

TCP/IP

Stack-2

LPAR-1

TCP/IP

Stack-3

LPAR-2

zSeries CPC-1

HiperSocketsIUTSAMEH

TCP/IP

Stack-4

LPAR-3

CPC-2

XCF Signaling

Dynamic connectivity is automatically established between all sysplexmembers

Using XCF messaging, each stack automatically learns of other sysplexmembers

Under-the-covers DYNAMICXCF interfaces are created using one of three transport technologies depending on availability and location of partner stack

– Inside same LPAR: IUTSAMEH (memory-link inside a z/OS system)

– Inside same CPC: HiperSockets (if enabled for that purpose via IQDCHPID VTAM start option)

– Between CPCs – XCF Signaling


Coupling

Facility

Sysplex Dynamic VIPAs, Stack managed DVIPAs

LPAR1

CICS

TCPIP1

Advertise:

201.81.10.7

VIPADEFINE

255.255.255.0 201.81.10.7

Single System Image:

– Location of application transparent to clients

– DVIPA identifies the TCP application independently of which stack in the Sysplex the server is currently active on

LPAR2

CICS

TCPIP2

VIPABACKUP

200 201.81.10.7

LPAR3

CICS

TCPIP3

VIPABACKUP

100 201.81.10.7

DVIPA activation is managed by the stacks:

– VIPADEFINE statement - defines the primary owner of the DVIPA

– VIPABACKUP statement –defines the backup owner(s) of the DVIPA

• Highest Rank determines which stack takes over

– TCPIP2 Rank 200

– TCPIP3 rank 100

Coupling Facility

– XCF messaging between sysplex members

– Automatic notification of member changes


Coupling

Facility


LPAR1

CICS

TCPIP1

VIPADEFINE

255.255.255.0 201.81.10.7

LPAR2

CICS

TCPIP2

VIPABACKUP

200 201.81.10.7

LPAR3

CICS

TCPIP3

VIPABACKUP

100 201.81.10.7

Unplanned takeover

If TCPIP1 fails,

– DVIPA moves to TCPIP2 since it is highest ranked backup,

– TCPIP2 advertises the DVIPA

– TCPIP2 CICS application takes over workload

– TCPIP2 sends connection resets to client on behalf of stack 1 which reduces workload restart time Advertise:

201.81.10.7


Coupling

Facility


LPAR1

CICS

TCPIP1

VIPADEFINE

255.255.255.0 201.81.10.7

LPAR2

CICS

TCPIP2

VIPABACKUP

200 201.81.10.7

LPAR3

CICS

TCPIP3

VIPABACKUP

100 201.81.10.7

Planned takeover

TCPIP1 is restarted

– DVIPA moves back to TCPIP1

– TCPIP1 advertises the DVIPA

– TCPIP1 handles routing for workload still going to TCPIP2 CICS

– TCPIP1 CICS application takes over new workload

Advertise:

201.81.10.7

V TCPIP,TCPIP1,SYSPLEX,DEACT,DVIPA=201.81.10.7

V TCPIP,TCPIP1,SYSPLEX,REACT,DVIPA=201.81.10.7

Operator commands allow planned takeover without changing configuration:


Application-specific DVIPAs

Single-instance applications require that one instance is active in the Sysplex:

– Application may need exclusive access to certain resources

The Application controls the activation and movement of the DVIPA:

– In the example, CICS1 initially causes the DVIPA to be created on stack 3 (Bind or MODDVIPA IOCTL)

– The application fails, and restarts on another LPAR (stack 4), causing the DVIPA to move from stack 3 to stack 4 (existing connections will be reset)

– Speed of recovery – controlled by ARM (Automated Restart Manager) or automated operations

TCPIP3

CICS1

10.1.1.1

Advertise:

DVIPA 10.1.1.1

TCPIP4

CICS1

10.1.1.1

Advertise:

DVIPA 10.1.1.1

CICS1 fails.

Restarts on Stack4

DNS

Resolve:

cics1.mycom.comEither:

1. Resolve cics1.mycom.com

2. Connect to returned address

Or

1. Connect to cached (or hardcoded)

address

10.1.1.1 is an application-specific DVIPA

VIPARANGE statement defines a subnet range

VIPARANGE 255.255.0.0 10.1.0.0

10.1.1.1


Advertise:

201.81.10.7

VIPADEFINE

255.255.255.0 201.81.10.7

VIPADISTRIBUTE

DISTMETHOD SERVERWLM

201.81.10.7 PORT 5000

DESTIP ALL

Sysplex Dynamic VIPAs, Distributed DVIPAs

LPAR1

CICS

5000

TCPIP1

LPAR2

TCPIP2

LPAR3

TCPIP3

LPAR2

TCPIP2

VIPABACKUP

200 201.81.10.7

Single System Image:

– Multiple applications look like a single application to clients

– DVIPA identifies the TCP application independently of which stacks the application is currently active on

Sysplex Distributor:

– Balances Workload within Sysplex by using a distributed DVIPA, advertised by the distributing stack

Datapath - either

– XCF interfaces or VIPAROUTE

VIPAROUTE

– IP route from distributor to target

– Allows OSA instead of XCF

– Doesn’t use CF resources

Sysplex

Distributor

CICS

5000

CICS

5000

VIPADISTRIBUTE statement

– specify DVIPA (and PORT(s)) of listening applications

• Dynamic Ports – Learn port as application starts a Listener

– DISTMETHOD - distribution method

– DESTIP – XCF IP address of each target stack or ALLstacks


Sysplex Distribution methods

The sysplex distributor uses one of five distribution methods when routing incoming connections to TCP/IP stacks:

– Static Distribution:• RoundRobin – Evenly distribute connections to all targets• WeightedActive – Balance active connections to all targets based on configured weight

proportions

– Distribution using dynamic WLM recommendation:Weighted round-robin distribution based on a normalized WLM weight:• BaseWLM – WLM system weights

– Comparison of displaceable CP capacity of lower importance work

• ServerWLM – WLM server-specific weights– Comparison of displaceable CP capacity given importance level of workload– How well is application meeting their WLM service class goals

– HotStandby – one active target, multiple backups

10 Connection Requests

3 Connections

2 Connections

XCF1

XCF2

Distributor

XCF35 ConnectionsXCF2 XCF3

XCF1

Normalized

Weight

25

3Ready 1Ready 1

Ready 1

Distribution using WLM recommendations:


Sysplex Distribution using WLM – additional monitoring

TSR: Target Server Responsiveness fraction, which is a compound health-metric per target server (range from 0 (bad) to 100 (good)):

– TCSR: Target Connectivity Success Rate. Connectivity between the distributing stack and the target stack - are the new connection requests reaching the target? (0 is bad, 100 is good)

– SEF: Server accept Efficiency Fraction. Target Server accept efficiency - is the server accepting new work? (0 is bad, 100 is good)

– QoS: QoS fractions. Consider retransmits and packet loss. (0 is good, 100 is bad)

– Number of half-open connections also impact the TSR value

CER: Connection Establishment Rate. Network connectivity between Server and client - are new connections being established? (0 is bad, 100 is good)

– Does not impact the TSR value, but is included in netstat displays


Sysplex Distribution - Weighted Active

10 Connection Requests

3 Connections

2 Connections

XCF1

XCF2

Distributor

XCF35 Connections

Initial Distribution

Active Conns

XCF2 20 0 2XCF3 50 0 5

XCF1 30 0 3

VIPADEFINE DVIPA1

VIPADISTRIBUTE DISTMETHOD WEIGHTEDACTIVE

DVIPA1 PORT 6000

DESTIP XCF1 WEIGHT 30

XCF2 WEIGHT 20

XCF3 WEIGHT 50

Active connections balanced across ready targets

10 Connection Requests10 Connections

XCF1

XCF2

Distributor

XCF3

Over Time

Active Conns

XCF2 20 190 200XCF3 50 530 530

XCF1 30 300 300

ReadyReady

Ready

ReadyReady

Ready


Sysplex Distributor - Hot Standby support

A single target server receives all new connection requests

– Other target servers are active but not receiving new connection requests

– Automatically route traffic to “hot-standby” target server when the active target server is not available

Enable using a new HOTSTANDBY distribution method

– One preferred target

– And one or more backup targets ranked in order of preference

– A target is not available when:

• Not ready

• Route to target is inactive

• If HEALTHSWITCH option configured – target is not healthy when

– TSR = 0% OR

– Abnormal terminations = 1000 OR

– Server reported Health = 0%

– AUTOSWITCHBACK option - switch to the preferred target if it becomes available

• No auto switch back if reason for original switch was health problems

– Use a V TCPIP Quiesce and Resume sequence

Sysplex

Distributor

Preferred

Target

server

Backup

Target

server

DB2 DB2

DB2 Data

sharing group

VIPADEFINE DVIPA1

VIPADISTRIBUTE DISTMETHOD HOTSTANDBY

AUTOSWITCHBACK HEALTHSWITCH

DVIPA1 PORT nnnn

DESTIP XCF1 PREFERRED

DESTIP XCF2 BACKUP 50

DESTIP XCF3 BACKUP 100


Sysplex Distribution – ServerWLM Importance level weighting

Server-specific weights

Comparison of displaceable capacity given the importance level of the service class.

Importance

level

0

1

2

3

5

4

6

7

LPAR2

0 Service units

0 Service units

0 Service units

0 Service units

500 Service units

LPAR1

0 Service units

0 Service units

0 Service units

0 Service units

500 Service units

New JAVA work

Importance level 2

Displaces

Level 3-7 Workloads

Example:

WLM weights – same for LPAR1 and LPAR2 – Same displaceable level 3 – 7 SUs, 500 SUs

But LPAR2 (500 SUs IL 5) should be favored over LPAR 1 (500 SUs IL 3)

ILWEIGHTING parameter - favor LPARs with lower importance level displaceable SUs – IL 0 Ignore – no change from existing logic. – IL 1 Moderate – IL 2 Aggressive – IL 3 Exceptionally Aggressive

VIPADISTRIBUTE


ILWEIGHTING 1


WLM weights and specialty processors

Distribution also considers specialty processors:

Specialty processors are deployed and exploited by targeted workloads on z/OS:

zAAP (zSeries Application Assist Processor) – JAVA workloads such as WAS

zIIP (System z Integrated Information Processor)

– DB2 related workloads

– z/OS IPSEC workloads


Sysplex Distribution – ServerWLM Crossover CostServer-specific weights

Comparison of displaceable capacity based on actual usage percentage of each processor by the workload

Example:

JAVA workload designed to use 10% CP 90% zAAP SUs

Each LPAR has same displaceable capacity - 1000 SUs

But as 1000 SUs are consumed - ideally 100 CP and 900 zAAP SUs consumed

PROCXCOST parm penalizes zAAP intended SUs that crossover and run on the CP

Higher PROXCOST value, the more WLM favors LPAR2

PROCXCOST ZAAP 1 ZIIP 1 – no penalty LPAR1 = LPAR2

VIPADISTRIBUTE


PROCXCOST ZAAP 5 ZIIP 5

Displaces

Level 3-7 WorkloadsNew JAVA work

Importance level 2

Designed Usage

CP 10% zAAP 90%Actual LPAR1 Usage

CP 90% zAAP 10%VS

LPAR2

CP SUs zAAP SUs

0

1

2

3

5

4

6

7

100 SUs 900 SUs

0 SUs 0 SUs

0 SUs 0 SUs

0 SUs 0 SUs

0 SUs 0 SUs

Importance

level

CP SUs zAAP SUs

900 SUs 100 SUs

0 SUs 0 SUs

0 SUs 0 SUs

0 SUs 0 SUs

LPAR1

0 SUs 0 SUs


Sysplex Distribution - Timed Affinity

LPAR1 LPAR2 LPAR4LPAR3

CICS1 CICS2 CICS4CICS3

TCPIP2 TCPIP3 TCPIP3TCPIP1

External Network

TIMEDAFFINITY– Choose target for new client using distribution method– Same target for new connections from same client IP@– Keep affinity until TIMEDAFFINITY seconds after last

connection ends

VIPADISTRIBUTE TIMEDAFFINITY 5– Keep affinity until 5 seconds after last connection ends.

Used with

– Base and Server WLM distribution– Weighted Active– Round Robin

201.81.10.1

Tier 1 SD

199.81.10.2


Sysplex Distribution – OPTLOCAL, Optimized local routing


CICS1 CICS2

Tier 1 SD

CICS4CICS3

Tier 2 Connection setup and Data path stay local when possible

Tier 2 SD

WAS1 WAS2 WAS4WAS3

Client Connections

VIPADISTRIBUTE OPTLOCAL

•Prefer local server

•Control level of preference

Sysplex Distributor can

optimize session setup

and datapath when

connection endpoints

are on the same TCP/IP

stack

•OPTLOCAL 0: Use local server

if there are no server problems:

not ready, backlog growing, abnormal transactions, server health

•OPTLOCAL 1: Use local server

if there are no server problems & WLM raw weight is not zero.

•OPTLOCAL 2-16: Use local server

if there are no server problems & WLM raw weight is not zero &

normalized weight * value > normalized weight of non-local servers


Use SD if OPTLOCAL

conditions not met

External Network


Sysplex Distribution - Tier 1 load balancing using Tier2 application awareness


CICS1 CICS2

Tier 1 SD

CICS4CICS3

WAS1 WAS2 WAS4WAS3


WLM 10 WLM 10 WLM 10 WLM 10

•Tier 1 distribution decision - based on

both application tiers

•Application tiers linked by group name

•The Tier 1 distributor chooses target

using combined weight of Tier1 and

Tier2 Group1 servers on each LPAR

Tier 1 GROUP1

Tier 2 GROUP1

T1T2 5 T1T2 10 T1T2 10T1T2 10

Tier 2 SD


VIPADEFINE TIER1

255.255.255.0 DVIPA1

VIPADISTRIBUTE TIER1 GROUP1


DVIPA1 Port 10000

DESTIP ALL

1

VIPADEFINE TIER2

255.255.255.0 DVIPA2


OPTLOCAL 1


DVIPA2 Port 10002

DESTIP ALL

2

2

1


Sysplex Distribution - Tier 1/Tier 2 CPC optimizations

CPC2

LPAR1 LPAR4LPAR3

Tier 1 SD

Tier 2 SD CPCSCOPE

WAS4WAS3

TCPIP3 TCPIP4TCPIP1

WLM 0 WLM 10

WLM 10 WLM 10

WLM 10

T1T2 5 T1T2 10 T1T2 10

CPC1

WAS1

CICS1 CICS4CICS3Tier 2 GROUP1

LPAR2

TCPIP2

T1T2 10

CICS2

Tier 2 SD CPCSCOPE

WLM 10 WLM 10

Tier 1 GROUP1

T2

GR

OU

P1

Wts

Client Connections

WAS2

WLM 10

Tier 2 GROUP1

VIPADEFINE TIER1

255.255.255.0 DVIPA1



DVIPA1 Port 10000

DESTIP XCF1 XCF2 XCF3 XCF4

1

VIPADEFINE TIER2 CPCSCOPE

255.255.255.0 DVIPA3


OPTLOCAL 1


DVIPA3 Port 10002

DESTIP ALL

2

CPCSCOPE – keep connections with Tier 2 SD’s CPC

VIPABACKUP DVIPA – within CPC

When Tier 1 SD chooses WAS1

– Tier 2 SD will choose CICS2 since CICS1 weight is 0

– Tier 2 connection path stays within CPC

• Better performance

• Potential to relax security requirements

2

1


Sysplex Distributor support for DataPower

CPC1


CICS CICS CICSCICS

WAS WAS WASWAS


IMS IMS IMSIMS

DB2 DB2 DB2DB2

Tier1:

CPC2

Sysplex Distributor Sysplex Distributor Web

Service

request

Sysplex Distributor

•DataPower weights for Load Balancing

•Connection state information allows

non-disruptive takeover, return path to

bypass SD

Tier1 SD TCP control connection to each DataPower

Distributor sends DVIPA/Ports & polling interval

DataPower

– Configures non-advertised DVIPAs

Sends:

– Server availability

– Connection states• Allows non-disruptive takeover

by backup SD

• Termination – allows GRE routing (bypass SD on return path)

– Weight (capacity)• SD load balances using

DataPower weights


Sysplex Distributor support for DataPower, Target Controlled distribution


Tier 1 SD

WAS1 WAS2 WAS4WAS3


Tier 1 GROUP1

Tier 2 SD


201.81.10.1

1VIPADEFINE TIER1

255.255.255.0 DVIPA1


GRE CONTROLPORT 1702

DISTMETHOD TARGCONTROLLED

DVIPA1 Port 10000

DESTIP 201.81.10.1 201.81.10.2

201.81.20.3 201.81.20.4Web

Service

request

1

201.81.10.2

DATAP1 DATAP2 DATAP3 DATAP4

201.81.20.3 201.81.20.4

Tier 1 SD supports DataPower:

TARGCONTROLLED distribution

GRE

– Target is non-z/OS target (DataPower)

– Routing type is GRE

CONTROLPORT

– destination port for TCP control connection


Solution: Sysplex Distributor Multi-Site support for DataPower

CPC1 CPC2


Tier 1 SD

WAS1 WAS2 WAS4WAS3


•Each DataPower - configured to

connect to a group of servers on a

site (CPC).

•Tier 1 distribution – load balance

using DataPower weight and site

weight of matching Tier 2 server

group (GROUP1)

•DATAP4 is preferred over DATAP3

based on reported datapower

weights.

•DATAP3 & DATAP4 are preferred

over DATAP1 and DATAP2 based

on TIER 2 server weights.

Tier 1 GROUP1

Tier 2 GROUP1


Web

Service

request

DATAP1

DP Wt 500

Tier 2 SD CPCSCOPE Tier 2 SD CPCSCOPE

DATAP2

DP Wt 500

DATAP3

DP Wt 500

DATAP4

DP Wt 900

NmWt 3 NmWt 3 NmWt 5 NmWt 9


SYSPLEXPORTS - Outbound Connections from Distributed DVIPAs

SYSPLEXPORTS:

– Ensure unique ephemeral source ports for each DVIPA in sysplex

– Coupling Facility - manages source DVIPA ports.

– Allocated from CF inblocks of 64 ports

– Used as server applications establish connections to clients

Client IP address 8.1.1.1 Port 3001

TCPIP2 connections from from DVIPA 10.1.1.1 uses source ports from 5001-5064 port range

TCPIP3 connections from DVIPA 10.1.1.1 uses source ports from 5068, 5070-5129 range

TCPIP1

TCPIP2

TCPIP3

Sysplex

Distributor

CICS

10.1.1.1

CICS

10.1.1.1

Advertise:

DVIPA 10.1.1.1

DVIPA 10.1.1.1

…

5001-5064

5068,5070-5129


SWSA – Sysplex Wide Security Associations

2. Shadow tunnel

created from target

back to client

1. Distributor handles

SA negotiation

Sysplex Distributor support for IP Sec workloads target systems

Distributor handles SA negotiation and refresh

Requires Coupling Facility DVIPA Takeover

–Keeps Tunnel renegotiation information

–Tunnel sequence number for shared tunnel

–SYSPLEXPORTS - ensure unique source ports for outbound connections from distributed applications back to the same client

TCPIP2

CICS

10.1.1.1

TCPIP3

CICS

10.1.1.1

TCPIP1

Sysplex

Distributor

Advertise:

DVIPA 10.1.1.1


XCF Monitor:

Is each Stack’s

sysplex component

responsive?

Sysplex Autonomics Stack Monitor:

OMPROUTE up?

VTAM up?

XCF links up?

Am I ok?

Connectivity to

router?

Stack Monitor:

OMPROUTE up?

VTAM up?

XCF links up?

Am I ok?

Connectivity to

router?

Stack Monitor:

OMPROUTE up?

VTAM up?

XCF links up?

Am I ok?

Connectivity to

router?

Sysplex Problem Detection and Recovery

GLOBALCONFIG SYSPLEXMONITOR TIMERSECS 60(NO)DELAYJOIN(NO)MONINTERFACE (NO)DYNROUTE(NO)RECOVERY (NO)AUTOREJOIN

Detect problems that affect TCP/IP Sysplex Health:

– Time dependent monitoring:

• Is OMPROUTE active?

• Is VTAM active?

• Is storage (common or private) critical?

• Are important interfaces (and routes over those interfaces) active?

• XCF detection – Are sysplex functions responsive?

– Non-time dependent monitoring – Sysplex abend

Recovery from problems:

–Leave sysplex group (RECOVERY)

–Issue actionable message

–Monitoring continues for recoverable problems

•OMPROUTE, VTAM, Storage, Interface (and route)

•When problem clears, clear message, rejoin sysplex group (AUTOREJOIN)


Subplexing

Subplexing allows isolation of multiple secure areas.

•Without subplexing:

SD requires DYNAMICXCF

–Establishes IP Connectivity to all stacks in Sysplex

–Dynamic IP@ discovery across sysplex

Coupling Facility resources span the sysplex

–VTAM generic resource, Multi-Node Persistent Sessions resources

–Sysplex ports resources, SWSA resources

LPAR2LPAR1 LPAR3 LPAR4 LPAR5

VTAM1

Internet SNA

TCPIP1

Intranet Production IP

Intranet Dev. IP

TCPIP2 TCPIP3 TCPIP4 TCPIP5TCPIP8

VTAM3 VTAM4 VTAM5VTAM2

TCPIP6 TCPIP7

Internet IP

Intranet SNA


LPAR2

Internet SNA subplex

VTAM XCFGRPID:Default (01)

LPAR1 LPAR3 LPAR4 LPAR5

VTAM2VTAM1

Internet IP subplex

TCP/IP XCFGRPID:Default (01)

TCPIP2TCPIP1

Intranet SNA subplex

VTAM XCFGRPID: 21

VTAM3 VTAM4 VTAM5

Intranet Production IP subplex

TCP/IP XCFGRPID: 02

TCPIP3 TCPIP4 TCPIP5

TCPIP8

Intranet Dev. IP subplex

TCP/IP XCFGRPID: 04

TCPIP7TCPIP6

Sysplex Group: EZBTCPCS Sysplex Group: EZBT2102

Sysplex Group: EZBT2104

SubplexingDYNAMICXCF

–Establishes IP Connectivity to all stacks in Subplex

–Dynamic IP@ discovery across subplex

Coupling Facility resources span the subplex

–VTAM generic resource, Multi-Node Persistent Sessions resources

–Sysplex ports resources, SWSA resources