remote journaling = better plumbing -...

IBM eServerJ iSeriesJ

8 Copyright IBM Corporation, 2003. All Rights Reserved.This publication may refer to products that are not currently available in your country. IBM makes no commitment to make available any products referred to herein.

Session:

Larry YoungrenAdam Stallman

Remote Journaling = Better Plumbing

Webcast

1

IBM eServer iSeries

© 2003 IBM Corporation

Receiver

Journal

Source System

SQL Table

Remote Receiver

Target System

HSL

Applications

Remote Journal

2

IBM eServer iSeries


What can we do with such technology ?

3

IBM eServer iSeries


Receiver

Journal

Remote Journal

SQL Table

Replicated SQL Table

Remote Receiver

Target SystemHa

rves

t

Applications

Source System

HA VendorReplay

Software

Continuous HA recovery

4

IBM eServer iSeries


Receiver

Journal

Source System

SQL Table

SQL Table

Remote Receiver

Target System

Harv

est

Applications

Role Swap

HA Vendor Software

Remote Journal

5

IBM eServer iSeries


Receiver

Journal

Source System

SQL Table

SQL Table

Remote Receiver

Target System

Harv

est

Applications

Role Swap

Remote Journal

Delta

Why the extra Journal on

the Target ?

6

IBM eServer iSeries


Better Plumbing ?

7

IBM eServer iSeries


Receiver

Journal

Source System

SQL Table

ReplicaSQL Table

BP Software

M I Boundary

Target System

Send

Applications

W/o Remote

Journaling

Harv

est

Log Space

Pre-RJ use of HA BP software for Harvesting on Source

8

IBM eServer iSeries


Primary System Backup System

MI

Apps RCVJRNE job

Exit program

Target job

Apply job

Replicated DB files

Log space

DB files JRN

Communications transport

Send entry

Receive entry

JRNReceiver

Vintage approach

9

IBM eServer iSeries


Primary System Backup System

MI

Apps Apply job

Replicated DB files

DB files JRN

Communications transport

Receive entry

RCVJRNE job

RemoteJRN

JRNReceiver

JRNReceiver

RJ-enabled approach

10

IBM eServer iSeries


Less CPU overhead than prior support

Shifts work to the backup/target machine Frees up CPU cycles on production/source machine

Puts packets on the wire faster

More efficient transport mechanism

Microcode to microcode transport (below MI)

Memory to memory transfer

DB images can be sent to target machine in real time

How Does Remote Jrn Differ From Prior HA Support?

11

IBM eServer iSeries


Net:Better Plumbing !

12

IBM eServer iSeries


Lakeview Technologies (MIMIX)

Vision Solutions (OMS)

Data Mirror Corp. (iCluster)

Maximum Availability (*NoMax)

iTERA (Echo2)

IBM's Data_Propagator product (for Data Warehousing)

Others pursing independent solutions for smaller shops

e.g. Sloan's TAA tools from Barsa Consulting

Traditional HA BP's(offer both RJ and non-RJ versions)

Employ Pure Remote Journal

Data Replication players

13

IBM eServer iSeries


An enabler

New plumbing for HA Vendor products NOT a replacement for HA Vendor products !

Someone still needs to replay the resulting remote Journal entries

What is Remote Journal?

14

IBM eServer iSeries


Setting it up

Adam Stallman

15

IBM eServer iSeries


No application changes required

The building blocks:

Can code to APIs

Can use CL Commands HA Vendor products will often orchestrate these steps for you

How Do I Set It Up?

16

IBM eServer iSeries


Three CL Commands

Add Rmt Jrn

Rmv Rmt Jrn

Chg Rmt Jrn

CL Commands

17

IBM eServer iSeries


ADDRMTJRN RDB(JOSNA) SRCJRN(TESTRJ/J1) TGTJRN(*SRCJRN) RMTRCVLIB(*SRCRCVLIB) RMTJRNTYPE(*TYPE1) MSGQ(QSYSOPR) DLTRCV(*NO)

CHGRMTJRN RDB(JOSNA) SRCJRN(TESTRJ/J1) TGTJRN(*SRCJRN) JRNSTATE(*ACTIVE) DELIVERY(*SYNC)STRJRNRCV(*ATTACHED)

Notice that you drive the whole shebang from the production sys

Examples

Defining the Target

The Handshake

18

IBM eServer iSeries


So.. I've got two separate

Journals on my Target machine ? (both a Rmt Jrn

and a regular/local Jrn). How can I keep from getting

confused ?

19

IBM eServer iSeries


Redirection: Using an Alt Lib

Comm Line

Remote Receiver

ALT_Lib

RJ:Redirected

Library New

Journal

Source System

SQL Table

Receiver

REG_Lib

Target System

Local Receiver

Target System

SQL Table

Replica

REG_Lib

20

IBM eServer iSeries


How do you redirect a remote journal ?

1. On target: CRTLIB LIB(ALT_LIB)2. ADDRDBDIRE RDB(RDBNAME) RMTLOCNAME(SYSTEM *IP)

where RDBNAME matches the *LOCAL RDB Entry on the Target and SYSTEM matches the Target's System Name or IP address

3. ADDRMTJRN RDB(RDBNAME) SRCJRN(REG_LIB/J1) TGTJRN(ALT_LIB/J1) RMTRCVLIB(ALT_LIB)

This command will create the remote journal in the Alternative (Redirected) library ALT_LIB on the target system.

4. CHGRMTJRN RDB(RDBNAME) SRCJRN(REG_LIB/J1) TGTJRN(ALT_LIB/J1) JRNSTATE(*ACTIVE)

This command will activate remote journal and create the journal receiver in library ALT_LIB on the target system.

Redirecting a Remote Journal

This will be your redirected (alternative) Library

Ahh... Name conflict resolution through

redirection !

21

IBM eServer iSeries


Three primary APIs

QJO Add Remote Journal

QJO Remove Remote Journal

QJO Change Journal State

Coding examples at:

http://www.as400.ibm.com/db2/db2code.htm

API Interfaces

Any coding examples ?

22

IBM eServer iSeries


How can I see the status of my Rmt Jrn

connection ?

23

IBM eServer iSeries


WRKJRNA - F16 Work with Remote Journal Information Journal . . . . . . : J1 Library . . . . . . : TESTRJ Journal type . . . . : *LOCAL Journal state . . . : *ACTIVE Remote journal type : Delivery mode . . . : Local journal . . . : Source journal . . . : Library . . . . . : Library . . . . . : ASP group . . . . : ASP group . . . . : System . . . . . . : System . . . . . . : Redirected receiver library . . . . . . . . . . . . . . . . . : *NONE Number of remote journals . . . . . . . . . . . . . . . . . . : 1 Type options, press Enter. 13=Activate 14=Inactivate ...

--------------------Remote-------------------- Relational Journal Delivery Opt Database Journal Library State Mode SYSTEM1 J1 TESTRJ *INACTIVE Bottom ===> F3=Exit F4=Prompt F5=Refresh F6=Work with remote journal list

F9=Retrieve F12=Cancel F23=More options

24

IBM eServer iSeries


Rmt Jrn uses RDB (Relational Database) entries to locate the remote systems

ADDRDBDIRE RDB(RDBNAME) RMTLOCNAME(SYSTEM *IP) Where RDBNAME matches the *LOCAL RDB Entry on the Target and SYSTEM matches the Target System's Name or IP address

Identifying the target machine So that's how Rmt

Jrn figures out which remote

machine to use !

25

IBM eServer iSeries


Six transport mechanisms: Opticonnect (Optical bus) Virtual (LPAR to LPAR) opticonnect TCP/IP SNA (APPC) ATM HSL

Chapter 6 of Journal Performance Redbook documents the most recent Remote Journal speeds over various communication lines

What Communication connections can I use ?

Everything but a tin can and a string

26

IBM eServer iSeries


Two Redbooks: Original Remote Journal Redbook:

AS/400 Remote Journal Function for High Availability and Data Replication

Found at www.ibm.com/redbooks Striving For Optimal Journal Performance

Remote Journal speeds in Chapter 6

Recent Magazine Articles: Eight "Tips" for Optimal Journal Performance

Jan and Feb '03 issues of iSeries News

ISeries Information Center: Found at www.ibm.com/eserver/iseries/infocenter

Resources

27

IBM eServer iSeries


Remote Journal Topologies

Larry

28

IBM eServer iSeries


Source Sys Target 2

Target 255

Target 1

Broadcast mode of Remote Journal

Can broadcast to up to 255 secondaries/targets

29

IBM eServer iSeries


Source Sys 1

Hub

Employing a centralized Regional Hub

Source Sys 2

Source Sys 3

30

IBM eServer iSeries


Source Sys

Second Target

First Target

Sync

Async

Cascade mode of Remote Jrn

Can cascade without limit

31

IBM eServer iSeries


Let's compare Transport modes

Sync Async

32

IBM eServer iSeries


Receiver

Journal

Source System

SQL Table

Remote Receiver

Target System

SLIC Async Sending Task

Async Remote

Journaling

Applications

Async Remote Journal Your

application doesn't wait

33

IBM eServer iSeries


Receiver

Journal

Remote Journal

SQL Table

Remote Receiver

Applications

Acknowledgment response

Target SystemSource System

Sync Remote Journal Your application pauses until the

response is received

34

IBM eServer iSeries


Sync mode

Big advantage: no trapped transactions All Journal entries reach the target system in real time

JRN image reaches target machine BEFORE reaching disk of source

Especially practical with Virtual OptiConnect, HSL, and high-speed ATM Most practical over short distances

Bene: No commit transactions are viewed as complete by your customer, then rolled back by your Take-over (target) machine !

Pros and Cons of Sync vs Async mode

Sync Async

35

IBM eServer iSeries


Async mode

Advantage: slightly less performance overheadProbably good choice for heavy batch jobs or slow communication linesCustomary choice when source and target machines are separated by substantial distances

Drawback: some risk of lost/trapped recent transactions not sent acrossSame kinds of risks as vintage async Business Partner solutionsIn-house for slow lines we've seen up to 1.5 sec latency

But only 5 Msec latency for virtual Opti-connect and HSL

Drawback: Probably tougher to recalibrate after a fail-over

Pros and Cons of Sync vs Async mode

Sync Async

Traffic Jam possible

36

IBM eServer iSeries


Ok, I see the HA benefit of Rmt Jrn.

But... what else can I do with this

technology ?

37

IBM eServer iSeries


Alternative uses of Remote Journal

Adam

38

IBM eServer iSeries


Receiver

Journal

Remote Journal

SQL Table

Target System

Remote Receiver

Applications

Source System

Alternative Rmt Jrn uses: Alternative #1 A remote Vault

Remote system:Less expensive.Head to Hot Site w/ tapes

39

IBM eServer iSeries


Production System

...

Alternate System

...

Remote Journal Connection

Saving from an alternate system reduces the impact on the production system. This also allows you to always have access to the latest changes to apply even if you haven't saved your Journal Receivers lately!

PF's

(large system $$$$$) (smaller system $)

(just the delta)

Off-Site Vaulting Example

40

IBM eServer iSeries


Receiver

Journal

Remote Journal

SQL Table

Data Warehouse

Target System

Harv

est

Remote Receiver

Applications

Source System

Alternative Rmt Jrn uses: Alternative #2Refresh a Data

Warehouse

Data Propagator

41

IBM eServer iSeries


Receiver

Journal

SQL Table

Target System

Remote Receiver

Remote Journal

Applications

Source System

Replaying from a Rmt JrnAlternative #3

Warm Site Hourly refresh

Replicated critical files

Local Receiver

ApyJ

rnCh

gX

Replay Step

How are you going to make the Rmt Jrn look like a "Local"

Jrn ?

42

IBM eServer iSeries


Rest

ore

Remote Receiver

Target System

Replaying from a Rmt JrnFooling Journal into thinking the

Rmt Jrn is a "Local"

Replicated critical files

Local Receiver

ApyJ

rnCh

gX

Save_File

43

IBM eServer iSeries


Can I APYJRNCHG from a Remote Journal Receiver ?

See online magazine article:MC Press online

1/21/03Remote Journaling and Data Recovery

by Chris Hird

Yes !

44

IBM eServer iSeries


Got any suggestions regarding how to

make Rmt Jrn have minimal overhead ?

45

IBM eServer iSeries


Remote Journal Performance Considerations

Larry Youngren

46

IBM eServer iSeries


What happens AFTERyour source machine is fixed ?

47

IBM eServer iSeries


Receiver

Journal

Source System

SQL Table

SQL Table

Remote Receiver

Target System

Harv

est

Catch-up phaseDelta

Delta

Swap Back to Production

HA Vendor Software

48

IBM eServer iSeries


The secrets of optimal Rmt Jrn performance

Watch that pesky extra

journal on the target machine

49

IBM eServer iSeries


Performance tuning on the Target side


50

IBM eServer iSeries


Speeding up Target side Remote Journal via Journal Caching

Secret #1

51

IBM eServer iSeries


User ASP

10

PF 3

Journal Bundling

Main Memory Buffer

IOA

Write Cache

128K Buffer

HA Replay Job #1

HA Replay Job #2

HA Replay Job #3

PF 1 PF 2

Journal Receiver

Journal

Known as a journal "Bundle"

Main Memory Journal Caching

Delta

52

IBM eServer iSeries


Trgt Jrn/ BP Async RJ Jrn Cache0

100200300400500600700800900

Thou

sand

sQ

ty tr

ansa

ctio

ns u

napp

lied

on T

arge

t

Shows worst case backlog unapplied on Target

Retail store Target backlogTeraplex model 840 w/ 11-CPU partition

HA BP software harvesting on Source vs. Rmt Jrn

Conclusion: Use of the Journal Caching PRPQ (5799-BJC) on the target machine helps HA Vendor replay software from falling behind during especially aggressive journal-intensive processing such as batch jobs.

290

793,601

565,518

Best case for Replay Only 0.1 Sec behind.

How far "behind" are the replay jobs on the Target ?

The Caching approach

53

IBM eServer iSeries


Potential Performance Benefits of Journal Caching

(For Target Machine's Keep-up Mode)

Apply rate on Target machine

Before Caching 600,000 transactions/Hr

With Caching on target 2,400,000 transactions/Hr

Source System

Target System

JrnCache(*Yes)

54

IBM eServer iSeries


Target machine suggestions:

Software (Journal) Caching

Enable Journal Caching on the Target machine

Elbow Room

Assure base pool on Target machine has enough elbow roomAt least 10 Meg for catch-up/reconnect phase per RJ connection

Sufficient Disk arms

Assure ASP housing target receiver has at least as many disk arms as matching receiver from the source system

Write cache

Configure lots of IOA write cache - - especially on the target machine

Choices: 26 Meg 104 Meg 757 Meg

What Else Can I Do To Speed It Up?

Newest

55

IBM eServer iSeries


Receiver

Journal

Need enough Arms

Write Cache

. . .

IOP 104 Meg of Write

Cache(older vintage IOPs

have less)

Don't skimp on the write cache Model Write Cache

(older) 2763 10 MegNEW !! 2782 40 Meg

(older) 2778 26 (104) MegNEW !! 2757 235 (757) Meg

Feb '03

Up to 1 Million Jrn Ents/Minute

56

IBM eServer iSeries


Have you got sufficient IOA Write cache ?

See Appendix "A"

Shows algorithm/Query to analyze write cache overflow

Found at www.ibm.com/redbooks

57

IBM eServer iSeries


Speeding up Remote Journal via Standby mode

Secret #2V5R2!

58

IBM eServer iSeries


Receiver

Journal

Remote Journal

SQL Table

SQL Table

HA vendor Software

Remote Receiver

Target System

Harv

est

Source System

Standby mode

CHGJRN... JRNSTATE(*STANDBY)

Standby mode

V5R2!

59

IBM eServer iSeries


Role Swap:

Receiver

Journal

Source System

ReplicatedSQL Table

HA Vendor Software

Active mode

Target System

Harv

est

Applications

Remote Receiver

Becomes Active

Switching out of Standby mode

CHGJRN ... JRNSTATE(*ACTIVE)

Delta

60

IBM eServer iSeries


BP/Trgt Jrn Async RJ Standby0

100200300400500600700800900

Thou

sand

sQ

ty tr

ansa

ctio

ns u

napp

lied

on T

arge

t

Shows worst case backlog unapplied on Target

Retail store backlog on Target Teraplex model 840 w/ 11-CPU partition

HA BP software harvesting on Source vs. Rmt Jrn793,601

Standby Mode

How far "behind" are the replay jobs on the Target ?

588

565,518

61

IBM eServer iSeries


Potential Standby mode Benefit:

Very little performance overhead on the target machine while in standby mode

Much more rapid failover in the event of a failureOS/400 merely flips one flag in the Journal rather than writing to disk thousands

of Database file headers

Mode transition rates during Role-swap as fast as...

How long does it take to re-enable Journaling on the Target during a

Role-swap ?

62

IBM eServer iSeries


10,000 files 25,000 files0

500

1000

1500

2000

2500

Seco

nds

Start Journaling (STRJRNPF) Standby to Active Transition(CHGJRN)

Start Journal vs. Standby to Active Transition

The V5R2 advantage

Role-Swap on the Target

V5R2!

63

IBM eServer iSeries


Summary

Net:

Order & Install Option 42

Option 42 encompasses both: JrnCache(*Yes) JrnState(*Standby)

V5R2!

64

IBM eServer iSeries


Performance tuning on the Source side


65

IBM eServer iSeries


Sending fewer bytes down the communication line(withholding the hidden entries)

Secret #3

66

IBM eServer iSeries


Main Memory

Journal Entry

AP

Journal Entry

PF

Write Cache

IOP2

Write Cache

IOP1

Hidden Jrn entries

Overlapped writes with CHGJRN ... RcvSizOpt(*RMVINTENT)

JOE

AP

User ASPOnly these must be

sent via RJ

. .

.JOE

PF

67

IBM eServer iSeries


Reduce the fixed overhead per Journal Entry(Sending only the data)

Secret #4

68

IBM eServer iSeries


Change Journal (CHGJRN) Type choices, press Enter. Journal . . . . . . . . . . . . > JRN1 Name Library . . . . . . . . . . . > JRNLIB Name, *LIBL, *CURLIB Journal receiver: Journal receiver . . . . . . . > *GEN Name, *SAME, *GEN Library . . . . . . . . . . Name, *LIBL, *CURLIB Receiver size options . . . . . *SAME *SAME, *NONE, *RMVINTENT... + for more values Minimize entry specific data . . *SAME *SAME, *NONE, *FILE, *DTAARA Journal caching . . . . . . . . *SAME *SAME, *NO, *YES Fixed length data . . . . . . . > *JOB *JOBUSRPGM, *JOB, *USR... > *USR > *PGM _ > *SYSSEQ + for more values > *THD Text 'description' . . . . . . . *SAME More...F3=Exit F4=Prompt F5=Refresh F12=Cancel F13=How to use this display F24=More keys

CHGJRN command with new FIXLENDTA parm

Customizing what we collect

V5R2!

69

IBM eServer iSeries


Summary

Be Selective:

Use the Goldilocks principle...

Advise us to collect only what you truly need, no more and no less

70

IBM eServer iSeries


Sending fewer bytes down the communication line(Sending only the delta)

Secret #5

71

IBM eServer iSeries


A new parameter (MINENTDTA) is provided to allow specifying which object types are allowed to minimize their journal entries.

Specified on: CRTJRN - Create Journal

CHGJRN - Change Journal

Journaling Minimal Data

72

IBM eServer iSeries


Journal Minimal Data is provided for Database Updates

With Journal Minimal Data enabled only the essential changed bytes are written to the journal (not the whole record image).

This can lead to:Reduction in the size of journal receiversSending minimized journal entries remotely for object replication will also reduce comm line traffic.

Journaling Minimal Data

The Remote Jrn advantage

73

IBM eServer iSeries


Interactive Job Batch Job

Reduction of Journal Receiver size

40% fewer bytes generated

60% fewer bytes generated

Performance consequences

4.6% more thruput,3.5% less CPU

3% reduction in batch run time

ESP Customer in Netherlands reported similar results

In-house measurement

74

IBM eServer iSeries


On the Target System:

Use Journal Caching CHGJRN ... JrnCache(*Yes)

Consider use of Journal Standby mode CHGJRN ... JrnState(*Standby)

On the Source System:

Employ CHGJRN ... RcvSizOpt(*RmvIntEnt)Puts a "fork" in the roadInhibits sending Access Path journal images to target system Do this only if you have more than one disk arm in your User ASP

Consider use of Journal-Minimal Data CHGJRN ... MinEntDta(*File)

Reduces quantity of traffic across the communication line Speeds up Re-play processing by HA vendor software

Consider use of MINFIXLEN or FIXLENDTA settings

Summary: Secrets to Good Rmt Jrn performance

75

IBM eServerJ iSeriesJ

8 Copyright IBM Corporation, 2003. All Rights Reserved.This publication may refer to products that are not currently available in your country. IBM makes no commitment to make available any products referred to herein.

Session:

Larry YoungrenAdam Stallman

Remote Journaling = Better Plumbing

Webcast

76

IBM eServer iSeries


Trademarks and Disclaimers

8 IBM Corporation 1994-2003. All rights reserved.References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

The following terms are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both:

Other company, product or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information in this presentation concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction.

Some information in this presentation addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here.

AS/400 IBMAS/400e IBM (logo)eServer iSeries

OS/400

77

remote journaling = better plumbing -...

Documents