quick and easy openedge ai file management · quick and easy openedge ai file management paul...

Quick and EasyOpenEdge AI File Management

PAUL KOUFALISPRESIDENT

PROGRESSWIZ CONSULTING

Progresswiz Consulting

Based in Montréal Québec CanadaBased in Montréal, Québec, CanadaProviding technical consulting in Progress®, UNIX, Windows, MFG/PRO and moreUNIX, Windows, MFG/PRO and moreSpecialized in performance tuning, system availability and business continuity planningy y p g…and security of Progress-based systems

© 2008 Progresswiz Consulting2 Quick and Easy OpenEdge AI File Management

Agenda

IntroductionIntroductionAfter-Imaging (AI) BasicsPre-OpenEdge Implementation ChallengesPre-OpenEdge Implementation ChallengesThe New AI File Management DaemonStep by Step ImplementationStep-by-Step ImplementationUnderstanding the ResultsA F Fi l StA Few Final StepsQuestions


Introduction

What is After-Imaging?What is After Imaging?

• Quick Definition:Quick Definition:

A logging system that stores all informationA logging system that stores all information required to reapply all changes made to a database between two points in time


Introduction

OpenEdge AI Gremlins Hard at Work!


Introduction

Recipe for RecoveryRecipe for Recovery• One backup• All AI files generated since that backupAll AI files generated since that backup

ResultResult• One fully recovered database with near-zero

data loss• “Near-zero” because open transactions will be

rolled back


Introduction

Why bother?Why bother?• Let’s see, because you like your job?

• Wait…wait…I know…data in a database is like, important, right?g


Introduction

But seriously folksBut seriously folks…• Most business’ only backup once per day• Without AI any changes made since the lastWithout AI, any changes made since the last

backup are lost

Best of all, it’s 100% free• Both Workgroup and Enterprise g p p• And Personal DB too!


After-Imaging (AI) Basics

Basic transaction process:Basic transaction process:1. Begin Transaction

a) Make some changeb) Before and after value written to BIc) Before and after value written to AId) Repeatd) Repeat…

2. End Transaction

Note that writes to the data files are not in the list• It will happen eventually



What happens to all these BI/AI notes?What happens to all these BI/AI notes?

• OpenEdge automatically manages the reuse• OpenEdge automatically manages the reuse of BI file space

• AI files must be archived before allowing OpenEdge to reuse them



Data Files BI Fil AI Fil 1 F llData Files(.d*)

BI File AI File 1 Full

D t

BI Notes Written

AI Notes Written

AI File 2

..

Full

Data Changes

AI File n

.

U P

Full

User Processes

Now What?



AI File 1EFull

AI File 1Empty

FullAI File 2

...

Empty Copy/Archive Safe Storage

AI File n

.

EmptyFull



Sounds simple enough:Sounds simple enough:• When full, copy/archive AI files to a safe

location• Mark them as empty so OpenEdge can reuse

the space in the files

What’s the problem?


Pre-OpenEdge Implementation Challenges

Problem #1: ScriptingProblem #1: Scripting

Commands are simple enough:Commands are simple enough:

• rfutil sports –C aimage extent full• rfutil sports -C aimage query extnum by name sports.a4p g q y y p• cp sports.a4 /archive/sports.${extnum}• rfutil <dbname> -C aimage empty

But adding the logic around them requires work



Scripts to automate AI file management canScripts to automate AI file management can be as simple or complex as you like• Depends how “smart” you want them to bep y

Reality: Not that difficultReality: Not that difficult• Plenty of example scripts available online• Search the peg (www.peg.com)Search the peg (www.peg.com)



Problem #2: Performance ImpactProblem #2: Performance Impact

Ye Olde Days of Yore yesYe Olde Days of Yore…yesToday: Please…• The effect is minimal unless you have really• The effect is minimal unless you have really

old junky hardware• Of course, there are exceptionsOf course, there are exceptions

– But if you’re that busy you really need AI!



Resist the “If it ain’t broke ” mentalityResist the If it ain t broke… mentality• If you are not running AI your system IS

broken

With modern hardware the performance issue t ode a d a e t e pe o a ce ssueis mootAnd now with the AI File Management gDaemon, so is the scripting issue


The New AI File Management Daemon

Available as of 10 1AAvailable as of 10.1AAutomates:• Rotation and reuse of AI files• Rotation and reuse of AI files• Archiving of full extents• The <ahem> intelligent <ahem> naming ofThe <ahem> intelligent <ahem> naming of

archived AI extents– More on this later

It’s about time!!!



Can be activated in one of two modes:Can be activated in one of two modes:• Time interval: AI file rotation occurs every x

seconds– Min: 2 minutes– Max: 24 hours

• Full AI File: Rotation occurs when the currently active AI file is filled

T i ll d ith fi d l th AI t t– Typically used with fixed length AI extents



Two configurable parameters:Two configurable parameters:• -aiarcdir: Directory where full AI extents will

be copiedp– Can specify multiple directories in case writes

to the first fail• -aiarcinterval: Time between AI file rotations

– If not specified, rotation occurs when extent fills


How it Works1. Daemon wakes up and checks for full AI files

or elapsed aiarcinterval

2. Full AI files are copied to the archive di t d d i t l

AI Daemondirectory and renamed appropriately

3. Once successfully copied, AI files are marked as empty and available for reuse

DB

AI Files ArchivedAI Files

BI File


Step-by-Step Implementation

Four ways to activate AI and AI File Mgmt:Four ways to activate AI and AI File Mgmt:

After-Imaging AI File ManagementOFFLINE OFFLINEONLINE OFFLINEOFFLINE ONLINE

N t O li ti ti f 10 1B

OFFLINE ONLINEONLINE ONLINE

Note: Online activation as of 10.1B


Add AI Extents

Can be done offline or online as of 10 1ACan be done offline or online as of 10.1ACreate add.st file# Three variable length AI files in /db/proda /db/proda /db/proda /db/prod

Apply to database$ prostrct add sports add.st$ prostrct addonline sports add.st


Enable AI & AI Mgmt Daemon Offline

Take a full backup of your databaseTake a full backup of your database$ probkup sports /backup/sports.probkp

Enable after-imaging$ rfutil sports –C aimage begin

Enable AI Management Daemon$ rfutil sports C aiarchiver enable$ rfutil sports –C aiarchiver enable


Enable AI & AI Mgmt Daemon Offline

Start your databaseStart your database• Be sure to include the –aiarcdir and, if

required, -aiarcinteval startup parametersq , p p

$ proserve sports –pf sports.pf –aiarcdir /aiarch /aiarch2aiarcdir /aiarch,/aiarch2-aiarcinterval 900


Enable AI & AI Mgmt Daemon Online

Enabling AI & the AI File ManagementEnabling AI & the AI File Management Daemon offline required three distinct steps:• Backupp• Enable AI• Enable AI File Managementg

As of 10.1B, do it all in one commandAs of 10.1B, do it all in one command


Enable AI & AI Mgmt Daemon Online

$ probkup online sports/backup/sports.probkpenableaienableaiarchi erenableaiarchiver-aiarcdir /aiarch,/aiarch2-aiarcinterval 900


Mixing Online and Offline

Parameters enableai and enableaiarchiverParameters enableai and enableaiarchiverare optionalCan use one or other in probkup online cmdCan use one or other in probkup online cmd• Of course, AI must be enabled in order to

enable AI Archiver


Mixing Online and Offline

Example:Example:• Backup offline

$ probkup sports /backup/sports.probkp

E bl AI ffli• Enable AI offline$ rfutil sports –C aimage begin

• Start DB$ proserve sports –pf sports.pf

• Enable AI Archiver online$ probkup online sports/backup/sports.probkp$ probkup online sports/backup/sports.probkp

enableaiarchiver-aiarcdir /aiarch,/aiarch2-aiarcinterval 900


Disabling the AI Archiver Daemon

To permanently disable the AI ArchiverTo permanently disable the AI Archiver$ rfutil sports –C aiarchiver disable

To stop the daemon temporarily without disabling the functionality• I.e. for maintenance$ rfutil sports –C aiarchiver end


Disabling the AI Archiver Daemon

No documented way to restart AI ArchiverNo documented way to restart AI Archiver• I.e. there is no “aiarchiver begin” • Will start automatically at next proserveWill start automatically at next proserve

• Or run the command:Or run the command:$ _mprshut sports -C aimgt


AI Archiver Status

Activated?Activated?• Use “proutil <dbname> -C describe

Database Features

ID Feature Active Details---- --------------------------------- ------ -------

8 After Image Mangement/Archiver Yes9 64 Bit DBKEYS Yes10 Large Keys Yes11 64 Bit Sequences Yes


AI Archiver Status

Running?Running?• Look in promon – R&D – 1 – 4 – 1 10/22/08 Stat s All Processes10/22/08 Status: All Processes7:17:24

Usr Name Type Wait Trans id Login timeyp g

0 Paul Kou BROK -- 0 0 10/22/08 17:095 AIMD -- 0 0 10/22/08 17:096 Paul Kou MON -- 0 0 10/22/08 17:16


Changing Parameters Online

Both parameters (-aiarcdir and –aiarcinterval)Both parameters ( aiarcdir and aiarcinterval) can be changed online

$ f til t C i hi tdi / i h$ rfutil sports –C aiarchiver setdir /aiarchnew

$ rfutil sports –C aiarchiver setinterval 1800

The second option is particularly useful• Decrease interval during high activityDecrease interval during high activity• Increase interval during quiet times


Understanding the Results

Enabling AI Archiver affects three outputEnabling AI Archiver affects three output streams:• DB Log Fileg• AI Archive output directory (-aiarcdir)• AI Archiver logg

– New


Database Log File

AI Archiver entries show up as “AIMGT”AI Archiver entries show up as AIMGTExample:[2008/10/22@17:27:16.078-0400] P-5316 T-912 I AIMGT 5: (13213) A new archive interval 120 has been set.

Please don’t go blind!AIMGT 5: (13213) A new archive interval 120 has been set.AIMGT 5: (13213) A new archive interval 120 has been set.


Archived AI Files

The good:The good:• All the information you need is stored in the

file nameThe bad:• All the information you don’t need is stored in y

the file nameThe ugly:C!~TEMP~oewrk~db~pk.20081022.164924.00002.00000017.pk.a2

• Now that’s a file name!


Archived AI Files

C!~TEMP~oewrk~db~pk.20081022.164924.00002.00000017.pk.a2

a b c d e f a b c d ea) The full file path of the source DBb) The date of the last backup) pc) The time of the last backupd) The sequence number of the last backup) q pe) The AI sequence numberf) The original AI file name


AI Archiver Log

AI Archiver creates it’s own log:AI Archiver creates it s own log:• <DBNAME>.archival.log

Whoa!! • A little bit challenging to read• A little bit challenging to read


AI Archiver Log

0001,C:\TEMP\oewrk\db\pk,20081022,181456,20081022,164924,2,36,C:\TEMP\oewrk\db\pk.a3,C:\TEMP\oewrk\db,C:\TEMP\oewrk\db\C!~TEMP~oewrk~db~pk.20081022.164924.00002.00000036.pk.a3

0032,C:\TEMP\oewrk\db\pk,20081022,181652,1,3,20081022,181652,38,1,pk.bkp.3,BACKUP SET 20081022 181652p.3,BACKUP_SET_20081022_181652

0001,C:\TEMP\oewrk\db\pk,20081022,181656,20081022,164924,3,37,C:\TEMP\oewrk\db\pk.a1,C:\TEMP\oewrk\db,C:\TEMP\oewrk\db\C!~TEMP~oewrk~db~pk.20081022.164924.00003.00000037.pk.a1

Black text is AI file rotation log entryBlue text is a probkup• Note the backup timestamp and sequence number

changed in AI file rotation entry


Careful!

AI Archive files and log show AI file #37AI Archive files and log show AI file #37 associated with backup #3:C:\TEMP\oewrk\db\C!~TEMP~oewrk~db~pk.20081022.164924.00003.00000037.pk.a1

Intuitively believe AI 37 should be applied to backup 3backup 3

– Incorrect– AI 37 is last AI file before backup 3AI 37 is last AI file before backup 3


Careful…continued…

From DB Log File:From DB Log File:Full backup started.Switched to ai extent C:\TEMP\oewrk\db\pk.a2. This is after-image file number 38 since the last AIMAGE BEGIN

Therefore AI file #38 is the first AI file to apply after restoring a backupafter restoring a backup• Not a problem• Rfutil utility will complainRfutil utility will complain

– “…expected file 38…”


Careful…continued…

UNC pathnames in -aiarcdirUNC pathnames in aiarcdir• I.e “-aiarcdir \\bkpserver\aidir”• There was a bug on WindowsThere was a bug on Windows• Not sure if fixed in 10.1C+


A Few Final Steps

AI Archive DirectoryAI Archive Directory• Ideally, directory mounted from another server

– NOT in the same building as productionNOT in the same building as production• Cleanup AI files older than x days

– Delete files older than x days after saving to y gtape

– 5 is good starting numberY ill h t i t thi lf– You will have to script this yourself


A Few Final Steps

MonitoringMonitoring• One more thing to monitor• mprshut <dbname> -C aimgt_mprshut <dbname> C aimgt

Log File Maintenance• <Dbname> archival log<Dbname>.archival.log

– Archive and truncate periodically


That’s It!

You’re done! Congratulations!You re done! Congratulations!

Restores and roll-forwards are easyRestores and roll-forwards are easy • AI Files are semi-intelligently named• AI Archiver log file shows which files go with• AI Archiver log file shows which files go with

which backupJust remember to cleanup AI ArchiveJust remember to cleanup AI Archive directory


Questions

Questions?Questions?


Progresswiz Consulting

Questions or comments? Send me an email:Questions or comments? Send me an email:[email protected]


quick and easy openedge ai file management · quick and easy openedge ai file management paul...

Documents