centera integration training
DESCRIPTION
Centera Integration Training. Advanced Centera Topics. Centera Corporate Systems Engineering Sept 2007. Agenda. Retention Management Advanced Retention Management features Retention Governors Event Based Retention Litigation Hold Extending Retention (non-EBR) - PowerPoint PPT PresentationTRANSCRIPT
1© Copyright 2007 EMC Corporation. All rights reserved.
Centera Integration Training
Advanced Centera Topics
Centera Corporate Systems Engineering
Sept 2007
Centera Integration Training: Advanced Centera Topics
2© Copyright 2007 EMC Corporation. All rights reserved.
Agenda
Retention Management– Advanced Retention Management features
Retention Governors Event Based Retention Litigation Hold
– Extending Retention (non-EBR)
Replication of Mutable Metadata
Content Addresses– Standard Formats– ClipID Canonical Form
SDK Logging
Wrap Up and Questions
Centera Integration Training: Advanced Centera Topics
3© Copyright 2007 EMC Corporation. All rights reserved.
Advanced Retention Management
Introduced in CentraStar / SDK 3.1
Separate licence
3 main features – Retention Governors– Event Based Retention– Litigation Hold
Centera Integration Training: Advanced Centera Topics
4© Copyright 2007 EMC Corporation. All rights reserved.
Enforced Retention
The presence of retention information can now be enforced– reject C-Clips without retention period or class.
Applies to pre-3.1 SDK only 3.1 SDK adds Default Retention Period if Application does not
– disabled by default
Valid retention value constraints can also be added– Min / Max retention Governors set on CentraStar– SDK enforces RetentionPeriod values to fall within the range
Centera Integration Training: Advanced Centera Topics
5© Copyright 2007 EMC Corporation. All rights reserved.
Min / Max Retention Governors Applied on Write, Replication and Restore
– Only applies to Retention Period values– C-Clips with a Retention Class are not checked against Min / Max– Not examined during Delete
Available on a per pool basis SDK Returns an error if value is outside the Min/Max
constraints:– FP_RETENTION_OUT_OF_BOUNDS_ERR
FPPool_GetCapability(FP_RETENTION, attribute) can check the limits
– FP_FIXED_RETENTION_MIN– FP_FIXED_RETENTION_MAX– FP_VARIABLE_RETENTION_MIN– FP_VARIABLE_RETENTION_MAX
Centera Integration Training: Advanced Centera Topics
6© Copyright 2007 EMC Corporation. All rights reserved.
Advanced Retention Management – Retention Governors
Application
SDK
CentraStar
Application creates C-Clip
SDK checks C-Clip
Retention InformationPresent?
SDK throws error
Server persists CDF and returns CA
Application deletes C-Clip
Retention Fits in Fixed Min/Max?
Apply defaultretention for the pool
(Infinite if CE+)
SDK writes C-Clip
Server deletes C-Clip
Centera Integration Training: Advanced Centera Topics
7© Copyright 2007 EMC Corporation. All rights reserved.
Advanced Retention: Default Values
Default values for all pools after enabling license:– Fixed retention minimum 0, maximum infinite– No enforced presence of fixed retention– Event based retention minimum 0, maximum infinite
Default Retention Periods must be within the Min/Max settings if present (except max in CE+, which can exceed it)
Centera Integration Training: Advanced Centera Topics
8© Copyright 2007 EMC Corporation. All rights reserved.
Event Based Retention (EBR)
Fixed retention can still be specified as before (period or class) – Must now comply with rules for Min/Max Governor
EBR can also be set on new C-Clips or on copies of existing C-Clips Disabled by default
– Enabled using FPClip_EnableEBRWithPeriod / Class Period must also comply with rules for Min/Max Governor
Stored in the C-Clip as before (immutable) No default value, but period or Class must be specified up-front
Centera Integration Training: Advanced Centera Topics
9© Copyright 2007 EMC Corporation. All rights reserved.
Event Based Retention (EBR)
EBR values applied when “triggered” once by a future event– FPClip_TriggerEBREvent uses the policy set when C-Clip created
New policy can be applied instead– FPClip_TriggerEBREventWithPeriod /Class– The longer of the old and new values will be used
New value only applied if longer than original setting
Must comply with rules for Min/Max Governor Event Date is set by SDK based on the cluster time Note: EBR information is not exported / imported by the Raw API
commands
Centera Integration Training: Advanced Centera Topics
10© Copyright 2007 EMC Corporation. All rights reserved.
Event Based Retention – Reporting
FPClip_IsEBREnabled:– Return TRUE if EBR is enabled, FALSE otherwise.
FPClip_GetEBREventTime:– Returns the time the EBR event was triggered– 0 if the event has not been triggered or the C-Clip is not EBR
enabled.
Centera Integration Training: Advanced Centera Topics
11© Copyright 2007 EMC Corporation. All rights reserved.
Event Based Retention – Reporting
FPClip_GetEBRPeriod:– Returns the greater EBR period associated with the clip
Initial value when C-Clip created / value set on Event trigger
– If EBR set with a class, the value of the class is used– 0 returned if the Clip is not EBR enabled.
FPClip_GetEBRClassName:– Returns name of the EBR retention class associated with the clip– Returns empty string if no class is set– … or if EBR period set on clip > period value of the class
Centera Integration Training: Advanced Centera Topics
12© Copyright 2007 EMC Corporation. All rights reserved.
Event Based Retention - Expiry
Deletion is possible when:– Event has occurred– AND Current time > creation date + fixed retention– AND Current time > event date + variable retention
Note: Privileged Delete can still overrides these rules.
C-ClipCreated
Time
Event
fixed retention
fixed retention
variable retention
variable retention
delete allowed
delete allowed
event not specifiedC1C1
event not specifiedC2C2
fixed retention
variable retentionevent not specifiedC3C3
delete allowed
Centera Integration Training: Advanced Centera Topics
13© Copyright 2007 EMC Corporation. All rights reserved.
Litigation hold
Litigation Hold(s) can be set / removed at C-Clip level– Retention is held indefinitely until removed.– Privileged Delete cannot override the Hold!
When a C-Clip is put under litigation hold via FPClip_SetRetentionHold the server tracks (outside the CDF):
– Litigation identifier – text string uniquely identifying the Hold– Litigation hold flag – true/false– Date/Time when the C-Clip was put under litigation hold.– Profile that put the C-Clip under litigation hold.
Centera Integration Training: Advanced Centera Topics
14© Copyright 2007 EMC Corporation. All rights reserved.
Litigation hold
C-Clip is released from a Hold by setting the hold flag to false
A C-Clip can be under multiple Litigation Holds at any time.
FPClip_GetRetentionHold returns TRUE if there are active Hold(s) on a C-Clip
Note: Cannot query for C-Clips that have active Hold(s) or obtain keys of C-Clips on hold
Centera Integration Training: Advanced Centera Topics
15© Copyright 2007 EMC Corporation. All rights reserved.
Advanced Retention Management – Event Based Retention
Application
SDK
CentraStar
Application creates C-Clip
SDK checks C-ClipSDK throws error
Server persists CDF and returns CA
Apply defaultretention for the pool
(0 if CE+)
SDK writes C-Clip
Application sets event
SDK checks event data
Variable Retention
InformationPresent?
Retention InformationPresent?
Retention Fits in Variable
Min/Max?
Retention Fits in Fixed Min/Max?
Server processes event
SDK sends event
Centera Integration Training: Advanced Centera Topics
16© Copyright 2007 EMC Corporation. All rights reserved.
Centera has a mechanism for placing a C-clip under fixed retention
Once a C-clip has been placed under fixed retention, the retention period cannot be modified
Customers may want to place a document under short term temporary retention, and then extend the retention period multiple times in the future
Note: this approach works well for extending the retention period– Decreasing retention may require Privileged Delete
To delete older CDF versions where retention has not yet expired
– Not allowed on CE+
Extending Retention without EBR
Centera Integration Training: Advanced Centera Topics
17© Copyright 2007 EMC Corporation. All rights reserved.
C-clip is created with a retention period / class– e.g. 30 days
Application stores the content address of the C-Clip– e.g. in local database
Extending Retention – Step 1 – C-Clip is created
blob
DQG22UC39LHJOe66R2K2JIBD4EF
Client Database
CDFretention = 30 days
Centera Integration Training: Advanced Centera Topics
18© Copyright 2007 EMC Corporation. All rights reserved.
At some point in time, retention requires to be extended– C-Clip is opened– New retention value is set in the CDF e.g. 60 days– C-Clip is written and the new ContentAddress is stored by the application
Replaces the original content address in the application database Standard SDK metadata in the new C-Clip contains old content address
Extending Retention – Step 2 – Retention is extended
DQG22UC39LHJOe66R2K2JIBD4EFEFG242769LH32e57R23E2IBC4FE
Client Database
blob
CDFretention = 30 days
CDFretention = 60 days
prev.clip=DQG22UC39LHJOe66R2K2JIBD4EF
Centera Integration Training: Advanced Centera Topics
19© Copyright 2007 EMC Corporation. All rights reserved.
Retention could be extended multiple times– EBR can only extend retention once
Extending Retention – Step 3 – Further extension of Retention
blob
EFG242769LH32e57R23E2IBC4FEE3423ABE3D5EBe34E3D32156A3a
Client Database CDFretention = 60 days
CDFretention = 7 years
prev.clip=EFG242769LH32e57R23E2IBC4FE
CDFretention = 30 days
Centera Integration Training: Advanced Centera Topics
20© Copyright 2007 EMC Corporation. All rights reserved.
C-Clip has had retention extended several times– “List” of CDFs to delete
Traverse the list deleting the CDFs– use previous clip ID as the “pointer” to the “ancestor” CDF
FPClip_GetDescriptionAttribute (clipID, “prev.clip”, outputBuffer, size)
– Original CDF has empty value for previous Clip ID
Blob is now unreferenced and will be Garbage Collected
Extending Retention – Step 4 - Deletion
blobCDF
retention = 60 days
CDFretention = 7 years
prev.clip=EF242769LH32e57R23E2IBC4FE
CDFretention = 30 days
prev.clip=DQG22UC39LHJOe66R2K2JIBD4EF
prev.clip=
Centera Integration Training: Advanced Centera Topics
21© Copyright 2007 EMC Corporation. All rights reserved.
Replication of Mutable Metadata
Replication will replicate any change in EBR and LH information
Profile used for replication needs several capabilities– clip-copy (c) for regular replication– delete (d) for global delete– privileged-delete (D) global privileged delete– clip-copy (c) for EBR information– litigation-hold (h) for LH information
Centera Integration Training: Advanced Centera Topics
22© Copyright 2007 EMC Corporation. All rights reserved.
Content Address Formats
Naming Scheme SizeSingle instance
Performance Enhanced?
Enhanced Collision Resistance?
M (MD5) 27 Yes No No
MG (MD5+GUID) 53 No No Yes
GM (w/o Discriminator) 53 No Yes No
GM+D (GM with Discriminator)
53 No Yes Yes
M++ (MD5 + 128b SHA-256) 53 Yes No Yes
SDK Collision Avoidance Disabled SDK Collision Avoidance Enabled
Cluster Storage Strategy Clip Blob Clip Blob
SSC (Capacity) M M (or M++) MG MG
SSPP (Performance Partial) M GM M GM+D
SSPF (Performance Full) GM GM GM+D GM+D
From 2.4 CentraStar, MG and GM forms are no longer used.
Centera Integration Training: Advanced Centera Topics
23© Copyright 2007 EMC Corporation. All rights reserved.
ClipID Canonical Format
Binary encoding for Content Address
Enables independence from character encoding specific to, e.g., Mainframe and Open Systems
– Content addresses generated by Mainframe use EBCDIC encoding!
Reduces byte count of CA from 65 bytes to 40 bytes– Handy for saving space, e.g., in a database record– Is not NULL-terminated
FPClipID_GetCanonicalFormat (const FPClipID inClipID, FPCanonicalClipID outClipID)
FPClipID_GetStringFormat (const FPCanonicalClipID inClipID, FPClipID outClipID)
Centera Integration Training: Advanced Centera Topics
24© Copyright 2007 EMC Corporation. All rights reserved.
SDK Logging
Highly configurable by category– Debug messages / API calls / Warnings / Errors
… component– POOL, RETRY, XML, API, NET, TRANS, PACKET, EXCEPT– REFS, MOPI, STREAM, CSOD, CSO, MD5, SHA– APP (for application level logging)– LIB (for library loading)
and file properties– Format / Path / Size / Overflow (backups)
Centera Integration Training: Advanced Centera Topics
25© Copyright 2007 EMC Corporation. All rights reserved.
Enabling SDK Logging with Configuration File
FPLogState.cfg file in application directory– Default method– Behaviour is dynamically controllable via changes in file state
Enable by adding file to directory Disable by deleting file from directory Alter by editing existing file in directory
– Polling interval is configurable (default is 5 minutes)
FP_LOG_STATE_PATH environment variable– SDK tests for this setting only if default configuration file not found– determines non-default location for configuration file
Behaviour is then the same as above
Centera Integration Training: Advanced Centera Topics
26© Copyright 2007 EMC Corporation. All rights reserved.
Enabling SDK Logging via Environment Variables
FP_LOGPATH – location of the output file
FP_LOGKEEP – creation mode– CREATE / APPEND / OVERWRITE
FP_LOGLEVEL – category of messages to log– FP_LOGLEVEL_ERROR / WARN / LOG / DEBUG
FP_LOGFILTER – components to log messages from
FP_LOGFORMAT – XML or TAB formatting
FP_LOG_DISABLE_CALLBACK– Disable the use of any application supplied logging callback method
FP_LOG_MAX_OVERFLOWS – number of backup logs
FP_LOG_MAX_SIZE – maximum size of log file
FP_LOG_STATE_POLL_INTERVAL– Polling frequency in minutes, default=5 (-1: off, 0: on each log event)
Centera Integration Training: Advanced Centera Topics
27© Copyright 2007 EMC Corporation. All rights reserved.
Configuring SDK Logging using API Calls
FPLogStateRef object holds current status– Get / Set methods change the property values– Serializable i.e. can be saved to or loaded from file
API calls to change logging status dynamically– FPLogging_Start(FPLogStateRef)– FPLogging_Stop()
Application can also log messages to the SDK logfile– FPLogging_Log(const FPLogLevel, const char *)
… or register its own logging method for the SDK to use– FPLogging_RegisterCallback(FPLogProc)– Can be disabled if required
Centera Integration Training: Advanced Centera Topics
28© Copyright 2007 EMC Corporation. All rights reserved.
Questions?
Please Visit SDK Forums on Centera Developer’s Portal– http://lighthouse.emc.com
Corporate Systems Engineering Team provides:– Info, Training, Whitepapers– Design Advice / Design Reviews– Debugging Support– Code Reviews
Email Support Available from:– [email protected]