ops-24: success with openedge® replication what you don’t know can hurt you cyril gleiman...

OPS-24: Success with OpenEdge® Replication

What you don’t know can hurt you

Cyril GleimanPrincipal Technical Support Engineer

© 2008 Progress Software Corporation2 OPS-24: Success with OpenEdge Replication

Obligatory (gratuitious) Quotation

An ounce of prevention is worth a pound of cure.

28.35 grams of prevention is worth .45 kilograms of cure


Success With Replication

Understand the architecture Know your requirements Analyze your system Plan, configure, deploy,monitor Succeed


Agenda

Terminology and architecture – Visual Replication performance hotspots Replication availability hotspots Source and target management


Replication Terminology

Primary – original production machine Secondary – original recovery machine Source – production database Target – recovery database Replication server – rpserver (source) Replication agent – rpagent (target) DBService queue – stores IPC messages Network pipe – TCP/IP rpserver -> rpagent

© 2008 Progress Software Corporation6 OPS-24: Success with OpenEdge Replication!

Architecture of Replication – Primary/Source

Primary Machine

R/WriteClients

AI Extents

DB Service

Q

Repl Server

AI ExtentsSource

Database

Database Brk/Server Processes


Architecture of Replication – Secondary/Target

Secondary Machine

Repl Agent

Database Brk/ServerProcesses

TargetDatabase

R/OClients


Architecture of Replication

Primary Machine Secondary Machine

Repl Agent


TargetDatabase

R/OClients

R/WriteClients

AI Extents

DB Service

Q

Repl Server

AI ExtentsSource

Database



Agenda

Terminology and architecture - Visual Replication performance hotspots Replication availability hotspots Source and target management


Performance Hotspots


Repl Agent


TargetDatabase

R/OClients

R/WriteClients

AI Extents

DB Service

Q

Repl Server

AI ExtentsSource

Database



Source DB Performance Considerations

Speed/power of machine Replication Plus – offload read only clients to

target database AI files – fixed extents

AI/BI blocksizes should be the same• 16 K is the most efficient

• Need to truncate ai and bi


Target DB Performance Considerations

System should not be underpowered by comparison to the source system

No need for after-imaging Read only clients


Network Performance

WAN vs. LAN The bigger the pipe the better Determine size of pipe Whitepaper available on PDSN http://tinyurl.com/6xqp78


Sample Network Bandwidth Calculation

Hourly After Image Blocks from 5 production databases = 713mb

By calculation:• Replication size = 1.5 * AI size

– 713mb*1.5 = 1069.5mb

• Add replication overhead (1.1)– 1069.5mb*1.1 = 1176.45mb

Throughput per second:• 1176.45mb/3600=.32679mb/b

Change to kilobytes /s:• .32679mb/s * 1000 = 326.79kb/b needed to sustain

transfer


Replication Performance – DBService Buffer

DBService Buffer Queue must be sized appropriately – Solution P121969• Failure to do so may result in source

slowdown and target falling behind

Analyze AI activity prior to implementing• Use PROMON -> R&D -> Status Display (#1)-

> DB Service Manager (#16).


Determining Optimal -Pica

The larger the –pica, the further behind the target may get

Monitor AI writes during busiest period for one hour

-pica = (TAIW/Blockcount) * 1.25 Blockcount depends on version

• OpenEdge 10 – blockcount is 9.16

• Progress® 9.1x – blockcount is 18.2


Sample –Pica Calculation

Values up to 8192k for 10.1B01 and later 34560 TAIW over one hour Formula for OE 10:

• (34560/9.16) * 1.25 = 4716.1572052401746724890829694323

Why not just use 8192???


“Houston, we have a problem” - Promon

03/03/08 Status: Database Service Manager

17:15:02

Communication Area Size : 2049.00 KB

Total Message Entries : 18733

Free Message Entries : 4

Used Message Entries : 18729

Registered Database Service Objects

Name Rdy Status Messages Locked by

OpenEdge Replication Server Y RUN 18729

OpenEdge RDBMS Y REG 0

OpenEdge DB Agent Y RUN 0


Agenda

Terminology and Architecture - visual Replication performance hotspots Replication availability hotspots Source and target management

© 2008 Progress Software Corporation20 OPS-24: Success with OpenEdge ReplicationCOMP-10: OpenEdge® Management & Replication – Divide et impera!

Availability Hotspots


Repl Agent


TargetDatabase

R/OClients

R/WriteClients

AI Extents

DB Service

Q

Repl Server

AI ExtentsSource

Database


© 2008 Progress Software Corporation21 OPS-24: Success with OpenEdge ReplicationCOMP-10: OpenEdge® Management & Replication – Divide et impera!

Availability Hotspots

Secondary Machine

Repl Agent


TargetDatabase

R/OClients

R/WriteClients

AI Extents

DB Service

Q

Repl Server

AI ExtentsSource

Database


DOWN!

Severed

Primary Machine Loss of Database TCP or Process Failure

Locked


Availability Hotspots – AI Management

Replication DOES NOT manage AI files AI Files must be emptied and backed up

• 10.1A AI archiver became available.

Database crashes when no empty AI extents.


Availability Hotspots – Locked AI Files

AI “locked” when AI notes not replicated to the target db• Common issue – often caused by

simple maintenance routines and failure to monitor status of replication

Configure the rpserver and rpagent to detect and handle outages

All about the *.repl.properties file!! Configure, test well and sleep easier at night


Availability – Many Locked AI Files

Use dsrutil monitor against source and target db. Check for replication shared memory

If rpagent is running, restart the replication server on the source

If rpagent is not running, restart target database and restart replication server


“Houston, we have a problem” – part 2

Extent: 1 Extent: 3Status: Busy Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a1 Path: C:\wrk101c\repl\source.a3 Size: 3192 Size: 6264 Used: 3180 Used: 6200 Start: Wed May 14 14:09:34 2008 Start: Wed May 14 14:08:04 2008 Seqno: 5 Seqno: 3

Extent: 2 Extent: 4Status: Locked Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a2 Path: C:\wrk101c\repl\source.a4 Size: 230008 Size: 25208 Used: 229674 Used: 25063 Start: Wed May 14 13:45:30 2008 Start: Wed May 14 14:08:24 2008 Seqno: 2 Seqno: 4


Availability – All AI Files Locked

Source database activity comes to a halt Can you afford to bring the db down?

• If not - disablesitereplication

• If yes – stop database– add new ai files– prostrct reorder– restart.


Prostrct add sourcedb addai.st

Extent: 1 Extent: 4Status: Busy Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a1 Path: C:\wrk101c\repl\source.a4 Size: 3192 Size: 25208 Used: 3180 Used: 25063 Start: Wed May 14 14:09:34 2008 Start: Wed May 14 14:08:24 2008 Seqno: 5 Seqno: 4

Extent: 2 Extent: 5Status: Locked Status: Empty Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a2 Path: C:\wrk101c\repl\source.a5 Size: 230008 Size: 120 Used: 229674 Used: 0Start: Wed May 14 13:45:30 2008 Start: N/ASeqno: 2 Seqno: 0

Extent: 3 Extent: 6Status: Locked Status: Empty Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a3 Path: C:\wrk101c\repl\source.a6 Size: 6264 Size: 120 Used: 6200 Used: 0Start: Wed May 14 14:08:04 2008 Start: N/ASeqno: 3 Seqno: 0


Prostrct reorder ai sourcedb

Extent: 1 Extent: 4Status: Busy Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a1 Path: C:\wrk101c\repl\source.a4 Size: 3192 Size: 230008 Used: 3180 Used: 229674Start: Wed May 14 14:09:34 2008 Start: Wed May 14 13:45:30 2008 Seqno: 5 Seqno: 2

Extent: 2 Extent: 5Status: Empty Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a2 Path: C:\wrk101c\repl\source.a5 Size: 120 Size: 6264 Used: 0 Used: 6200Start: N/A Start: Wed May 14 14:08:04 2008 Seqno: 0 Seqno: 3

Extent: 3 Extent: 6Status: Empty Status: Locked Type: Variable Length Type: Variable Length Path: C:\wrk101c\repl\source.a3 Path: C:\wrk101c\repl\source.a6 Size: 1 20 Size: 25208 Used: 0 Used: 25063Start: N/A Start: Wed May 14 14:08:24 2008 Seqno: 0 Seqno: 4


Source and Target DBs Won’t Synchronize

Try restarting a few times. -Ma, -Mn, -n match? Changes to .properties files? If it continues to fail, contact support

• Source/target db log files

• AI files

• pmmgr.properties file– arguments=-logging 2


Agenda

Terminology and architecture - visual Replication performance hotspots Replication availability hotspots Source and target management


Managing Replication – DSRUTIL Monitor and Status

Not immediately obvious when a failure occurs

Proactively verify replication performance and status


DSRUtil Monitor

DSRUTIL source/target –C monitor• Attaches to replication shared memory.

– Cannot connect to replication shared memory. Status = -1

• Checks status of Server and Agent– Server status– Agent status


DSRUtil source –C monitor

OpenEdge Replication Monitor Page 1 Database: C:\wrk101c\repl\source

Database is enabled as OpenEdge Replication: Source

Server is: In Normal Processing Number of configured agents: 1 Delay Interval (current / min / max): 5 / 5 / 500 Recovery information: State: No recovery being performed

Agents needing recovery: 0

Agents connected: 0 Agents in synchronization: 0 Transition information: Type: Manual


DSRUtil target –C monitor (page 1)

Database: C:\wrk101c\repl\target

Database is enabled as OpenEdge Replication: Target

Agent: Name: agent1 ID: 1 Host name: State: Normal Processing Ready: Yes Critical: No Method: Asynchronous Agent is waiting for: Nothing Maximum bytes in TCP/IP message: 8500 Server/Agent connection time: Wed May 14 13:48:43 2008 Delay Interval (current / min / max): 5 / 5 / 500 Transition information: Type: Manual The last block received at: Wed May 14 13:51:14 2008 Activity information: Blocks received: 2084 Blocks processed: 2084


DSRUtil target –C monitor (page 2)

Blocks acknowledged: 0 Notes processed: 207774 Transactions started: 10269 Transactions ended: 10269 Synchronization points: 33 AI Block Information: Source RDBMS Block (Seq / Block): 2 / 2516 Last Processed Block (Seq / Block): 2 / 2494 Latency Information: Repl Server behind Source DB by: 1 second(s) Current Source Database Transaction: 13906 Last Transaction Applied to Target: 13793 Target Current as of (Target, Source): Wed May 14 13:51:13

2008, Wed May 14 13:51:13 2008 with delta of 000:00:00


DSRUTIL Status

DSRUTIL source/target –C status Return code indicates current state of

replication server or agent. Good for automated scripts Grep for return code Replication User Guide provides details


Managing Replication – OE Management

OpenEdge Management 10.1B02/3.1B02+ remotely monitor log files

10.1B02 AdminServer running on the remote system

Attach OE_DB_Replication Log File Ruleset to the log file monitor

Alert/email thrown when message violating rule set appears.


OE Management – Replication Log File Rule Set


Summary

Terminology and Architecture Replication performance hotspots Replication availability hotspots Source and target management


Success With Replication

Understand the architecture Know your requirements Analyze your system Plan, Configure, Monitor Succeed

© 2008 Progress Software Corporation41 OPS-24: Success with OpenEdge Replication Session ID: Session Title

For More Information, go to…

PSDN• www.psdn.com/library/kbcategory.jspa?categoryID=21• www.psdn.com/library/kbcategory.jspa?categoryID=334

Knowledge Centrum (esupport.progress.com)• 3.1B01/10.1B01 upgrade: P122926, P123418, P123420,

P123424, P123426, P123427, P123676• sizing –pica: P121969

Documentation• OpenEdge Replication 10.1C docs (www.psdn.com)• OpenEdge Management 3.1C docs (www.psdn.com)


Relevant Exchange Sessions

OPS-21: Managing multiple sites part time: Replication & OpenEdge Management Case Study


Questions?


Thank You

ops-24: success with openedge® replication what you don’t know can hurt you cyril gleiman...

Documents

progress software corporation

replication size

mbadd replication

target management

ai size713mb

source slowdown

source systemno

behindanalyze ai activity