IBM Mainframe-IntegrationMainframe Change Data Capture
Presented by
The current situation:
Mainframe
Applications
+
Databases
Mobile
Applications
+
Databases
Internet/Web
Applications
+
Databases
Workstation/Client-Server
Applications
+
Databases
The current situation:
• ADABAS• IDMS/DB• DB/2• IMS and DL/I• VSAM• DATACOM/DB• other…
MainframeApplications
+Databases
The current situation:
• SQL Server• Oracle• DB/2 UDB• MS Access• other…
WorkstationClient/ServerApplications
+Databases
The current situation:
• Sybase• MySQL• DB/2 UDB• ORACLE• other…
Internet/Web-
Applications
+
Databases
The current situation:
• MS SQL• MS Access• DB/2 UDB• ORACLE• other…
MobileApplications
+Databases
The current situation – Data exchange:
Data exchange problems
• Different data formats• Different data models • Large data volumes• Limited batch window• Requirement for up-to-dateness of information
tcVISION – Mainframe Change Data Capture
The Solution
Moving data...
→ as much as needed→ as little...→ as transparent...→ as flexible...→ as secure... AS POSSIBLE
tcVISION – Mainframe Change Data Capture
The new data exchange generation
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Basic questions:
1. How current must the information be?
Data exchange between source- and target-system should be ...
• continuous / in realtime• cyclic / interval based• event based
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Basic questions:
2. What are the attributes of the datasource?
The datasource attributes influence the choice for the data capture method ...
• Size of datasource• Format of datasource (Structure, DBMS)• Change frequency• Amount of changes and• available resources (CPU-power, network bandwidth)
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
• Usage of the DBMS logging capabilities
•IMS/DB, VSAM, DB/2, DL/I, ADABAS, IDMS, DATACOM
• Transfer of changed data in scheduled time frame
• Best for batch window
• Best for processing right after logfile creation
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
• Efficient transfer of entire databases
• Analyzis for data consistancy
• Best for „Initial Load“ prior to log processing
• Best for periodic mass data transfer
• One step data transfer
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
• Comparison of data snapshots
• Efficient transfer of changed data since last processing
• IMS/DB, DL/I, VSAM, DB/2, ADABAS, IDMS, DATACOM, Sequential files
• Flexible processing options (SORT etc.)
• Automatic creation of deltas by tcVISION
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
• Realtime capture of changed data
• Changes directly obtained from DBMS
• IMS/DB, VSAM, DB/2, ADABAS, IDMS, DATACOM
• Secure data storage even across DBMS restart
• Flexible propagation methods
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Up-To-Dateness of changed data
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Data volume
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Steps:
• Extraction of database logs
• Process meta information
• Process changes of selected tables
• Propagate changes
• Implementation into target database
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Steps:
• Read datasource
• Process meta information
• Selective and structured transfer
• Implementation into target databse
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Steps:
• Read datasource
• Read corresponding snapshot
• Determine changes
• Process meta information
• Propagate changed data
• Implementation into target database
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Steps:
• Capture changes in DBMS
• Transfer changes into „Pool storage“
• Parallel processing of changed data from the pool
• Propagation of data to the target system
• Implementation into target database
tcVISION – Mainframe Change Data Capture
Every method has been designed to meet special needs and requirements. An efficient and secure data synchronization is guaranteed with all methods.
Summary
The tcVISION Data Capture Methods
tcVISION – Mainframe Change Data Capture
Transfer and Propagation
Propagation to other mainframe systems
Propagation to Internet-databases
Propagation to mobile systems
Propagation to client server databases
tcVISION – Mainframe Change Data Capture
Transfer and Propagation
1. Propagation independent of data origin
2. Propagation to different target systems (even for the same data)
tcVISION – Mainframe Change Data Capture
Examples
Case 1: DB/2 Log processing and implementation into DB/2 UDB
tcVISION – Mainframe Change Data Capture
Examples
Case 2: DL/I Real-Time Capturing and implementation into a DB/2 UDB and VSAM
tcVISION – Mainframe Change Data Capture
Examples
Case 3: IDMS- Realtime Capturing and bi-directional synchronization with ORACLE, DB2/UDB and MS-SQL
tcVISION – Mainframe Change Data Capture
Examples
All cases are Customer implementations, that are currently in production.
SQLServer
DL/I IDMS
tcVISION – Mainframe Change Data Capture
Examples
Steps:
1. Import of the metadata
2. Assignment of target tables
3. Definition of processing script
4. Test and implementation
tcVISION – Mainframe Change Data Capture
Examples
Import of meta data
The tcVISION Manager publishes the layout of the log-record for every table for later use by the processing scripts
tcVISION – Mainframe Change Data Capture
Examples
Assignment of target tables
Creation of target tablesor
Assignment of existing tables in the target system.
tcVISION – Mainframe Change Data Capture
Examples
Definition of a processing script
Use the tcVISION script wizard to create the script
Definition of all required parameter: Input and output, LUW processing etc.
tcVISION – Mainframe Change Data Capture
EXAMPLES
Test and implementation
Start and test the script
Implement script execution into scheduler and automatic processing
tcVISION – Mainframe Change Data Capture
Example
Technical aspects:
Standard data exchange was not possible due to extremely large amount of data.• 760 tables• 200.000 changes per day are processed• Propagation to DB2 UDB under SUN Solaris• Processing time with tcVISION per day less than 30 minutes
tcVISION – Mainframe Change Data Capture
Examples
Summary
Just a few steps to implement a data synchronization process
Solid monitoring and logging
High stability
Changes applied to structures of source databases are automatically processed
tcVISION – Mainframe Change Data Capture
Examples
Summary
Just a few steps to implement a data synchronization process
Solid monitoring and logging
High stability
Changes applied to structures of source databases are automatically processed
tcVISION – Mainframe Change Data Capture
Examples
Steps:
1. Define a collector and pool storage +Definition of a processing script
2. Creation of target table
3. Test and implementation
DL/I
tcVISION – Mainframe Change Data Capture
Examples
Creation of a collector/pool combination for the DL/I database
Definition of a processing script
tcVISION – Mainframe Change Data Capture
Examples
Define Collector as DBMS Extension
Define Pool to buffer the changes
Define processing script to implement into the target database
Use project wizard to specify the work-flow
tcVISION – Mainframe Change Data Capture
Examples
Test and implementation
Start the project using the tcVISION Control Board
tcVISION – Mainframe Change Data Capture
Examples
Technical aspects:
Standard data exchange was not possible due to extremely large amount of data.
• 40 DLI segments• 200.000.000+ records• 150.000+ changes per day • Implementation: Realtime using DBMS Extension Capturing of changes performed by Online- and Batch-programs.
tcVISION – Mainframe Change Data Capture
Examples
AdditionalImplementation:
If the target DB/2 is not available, changes are automatically saved into a VSAM KSDS
Highest possible recovery
Customer performs amainframe to mainframesynchronization
tcVISION – Mainframe Change Data Capture
Examples
Steps:
Step 1:
1. Create a collector/pool combination +Definition of a processing script
2. Creation of target table
3. Test and implementation
IDMS
SQLServer
tcVISION – Mainframe Change Data Capture
Examples
Import of the SCHEMA definitions for the IDMS database
SQLServer
tcVISION – Mainframe Change Data Capture
Examples
Define Collector as DBMS Extension
Define Pool to buffer the changes
Define processing scripts to implement the changes
Use the project wizard to define the necessary steps
SQLServer
tcVISION – Mainframe Change Data Capture
Examples
Test and implementation
Start the project using the tcVISION Control Board
tcVISION – Mainframe Change Data Capture
Examples
Technical aspects:
Conventional synchronization not possible• Bi-directional updates must be supported• Updates in DB2/UDB, ORACLE and MS-SQL are captured by TRIGGERS• TRIGGERS invoke tcACCESS/ODBC
to perform IDMS updates• tcVISION recognizes updates from tcACCESS and prevents a loop-back SQLServer
Mainframe
Applications
+
Databases
Mobile
Applications
+
Databases
Internet/Web
Applications
+
Databases
Workstation/Client-Server
Applications
+
Databases
tcVISION – Mainframe Change Data Capture
tcVISION – Mainframe Change Data Capture
Mainframe
Applications
+
Databases
Mobile
Applications
+
Databases
Internet/Web
Applications
+
Databases
Workstation/Client-Server
Applications
+
Databases
tcVISION – Mainframe Change Data Capture
• Different data formats• Different data models • Large data volumes• Limited batch window• Requirement for up-to-dateness of information
Data exchange challenges:
tcVISION – Mainframe Change Data Capture
• Automatic data conversion into the correct target format• Different data models • Large data volumes• Limited batch window• Requirement for up-to-dateness of information
Data exchange challenges:
tcVISION – Mainframe Change Data Capture
• Automatic data conversion into the correct target format• Adaptation of the data structure during transfer• Large data volumes• Limited batch window• Requirement for up-to-dateness of information
Data exchange challenges:
tcVISION – Mainframe Change Data Capture
• Automatic data conversion into the correct target format• Adaptation of the data structure during transfer• Only transfer changed data• Limited batch window• Requirement for up-to-dateness of information
Data exchange challenges:
tcVISION – Mainframe Change Data Capture
• Automatic data conversion into the correct target format• Adaptation of the data structure during transfer• Only transfer changed data• Batch-Compare with operational database• Requirement for up-to-dateness of information
Data exchange challenges:
tcVISION – Mainframe Change Data Capture
• Automatic data conversion into the correct target format• Adaptation of the data structure during transfer• Only transfer changed data• Batch-Compare with operational database• Real-Time Capturing meets maximum requirements for up-to-date information
Data exchange challenges:
tcVISION – Mainframe Change Data Capture
• Automatic data conversion into the correct target format• Adaptation of the data structure during transfer• Only transfer changed data• Batch-Compare with operational database• Real-Time Capturing meets maximum requirements for up-to-date information
The state-of-the-art demand forMainframe data exchange:
tcVISION – Mainframe Change Data Capture
YOU will profit from:
• Efficiency• Transparency• Auditability• Up-to-dateness
www.treehouse.com