2011 06 evolution of the ims database_nyc_toronto_ug
DESCRIPTION
Marilene Roder's presentation on the evolution of the IMS DB, Toronto IMS RUG meeting on June 30, 2011TRANSCRIPT
© 2011 IBM Corporation
Evolution of the IMS DatabaseFrom the Moon to the Cloud
Marilene Roder – Software Engineer
© 2011 IBM Corporation
Data Base
Concurrent Update
From Multiple
User/
System
Multiple Systems
Coupling (MSC)
Batch
DBMS
Exploit MP
Architecture
Two
Phase
Commit
Deadlock
Detection
> 2500 Transactions
Per Second
1968
2011+
Concurrent
Image Copy
Parallel
Systems
N-Way
Data Sharing2-way Data
Sharing
JDBC Access
Remote Site
Recovery
APPC/IMS
Distributed
Access
...
Web Services
> 1000 Transactions
Per Second
~28000 Transactions
Per Second
IMS Database Progress
Java
Dependent
Regions
© 2011 IBM Corporation
Agenda
� IMS Databases: overview
� Database Quiesce (IMS v11)
� Open Database (IMS v11)
� XML DB (IMS v9) and XQuery (IMS v10)
� Integration with other Tools
© 2011 IBM Corporation
IMS/DB is a Database Management System
� A Database Management System – is a collection of programs for storing, organizing, selecting, modifying, and
extracting data from a database.
� IMS DB is organized hierarchically– To optimize storage and retrieval – To ensure integrity and recovery– Segments are implicitly joined
A
B C DINSTRUCT
I 11
STUDENT
S 11
COURSE
C 1
CLASS
L 11
1213
1415
1617
12
13
© 2011 IBM Corporation
IMS Database Access Methods
Sequential Access
HSAM
SHSAM
GSAM
Indexed Sequential Access
HISAM
SHISAM
Direct Access Indexed Direct Access
HDAM
PHDAM
DEDB
MSDB
HIDAM
PHIDAM
© 2011 IBM Corporation
HALDB - What Is It?
� Large Databases– Databases are partitioned
• Up to 1001 partitions per database• Partitions have up to 10 data set groups• Partition definition separated from database structure definition
� High Availability Databases– Partition Independence
• Allocation, authorization, reorganization and recovery by partition– Self-Healing pointers
• Reorganization of partition does not require changes to indexes or logically related databases which point to the partition
� Partitioned Database Data Sets– PSINDEX
• 1 data set - secondary index data set– PHDAM
• 2 minimum/11 maximum - 1 for data, 1 for indirect list data set– PHIDAM
• 3 minimum/ 12 maximum - 1 for data, 1 for indirect data set, 1 for primary index data set
© 2011 IBM Corporation
IMS HALDB Highlights
� Database types
– PHDAM - partitioned HDAM
– PHIDAM - partitioned HIDAM• Index is also partitioned
– PSINDEX - partitioned secondary index
� Hierarchic structure is maintained
– A database record resides in one partition
� Partition selection
– By key range or by user exit routine
...
© 2011 IBM Corporation
Availability
Compatibility
Usability
Performance
IMS HALDB Benefits
� Greater database capacity
– "Unlimited" capacity
– Partitions may be modified as requirements change
� Increased database availability
– Shortened reorganization process
– Option of Online Reorg (IMS V9+)
– Batch window may be shortened with concurrent processing
� Improved performance– Concurrent processing against partitions
� Application program changes are not required
© 2011 IBM Corporation
Customer Examples –
� German Bank
– 600 GB total capacity of all
databeses, therefrom– 512 GB for all HALDBs
•Largest HALDB: 340 GB
� German Customer
– 17 HALDB
– Largest HALDB: 630 GB with 90
partitions and 2 secindex with 55
partitionen
� …
� France
– Avant 01/2004 en France•AGF, Euroinformation, Natixis,
Renault, Pole Emploi
– Après 2004
•GCA, …
� UK Insurance
– Largest HALDB: 140GB
© 2011 IBM Corporation
IMS HALDB Information
� Redbook:
– The Complete IMS HALDB Guide, All You Need to Know to Manage HALDBs
• SG24-6945 – June 2003
• http://www.redbooks.ibm.com/abstracts/sg246945.html?Open
� Presentations and articles on the Web
– IMS High Availability Large Database (HALDB)
• http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS185
– Migrating to IMS HALDB – Refreshed 2009
• http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS693
– Application Design and Programming with HALDB
• http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS490
– Using GENJCL.USER to Allocate IMS HALDB Data Sets
• http://www-1.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD100491
© 2011 IBM Corporation
�Integrated OnLine Reorganization of HALDBs
ƒ HALDB OLR provides 100% availability of the largest databases in the world!
ƒ OLR provides non-disruptive reorganization of HALDB PHDAM and PHIDAM partitions
�Concurrent IMS updates are allowed while OLR
is active
ƒ Planned data outage not required
Online HALDB Reorganization
© 2011 IBM Corporation
Relationship between Input and Output Datasets
© 2011 IBM Corporation
Agenda
� IMS Databases: overview
� Database Quiesce (IMS v11)
� Open Database (IMS v11)
� XML DB (IMS v9) and XQuery (IMS v10)
� Integration with other Tools
© 2011 IBM Corporation
Database Quiesce
� Offers the ability to stop accessstop access to a Database, HALDB Partition, DEDBDatabase, HALDB Partition, DEDB or Area,Area, or
Datagroup,Datagroup, allowing a singlesingle coordinated recovery pointcoordinated recovery point to be established
–– WithoutWithout taking the resource offline! No /DBR is required
• Type-2 UPDATE DB|AREA|DATAGRP command is used to start and stop quiesce
– The recovery point will be coordinated across the IMSplexacross the IMSplex to create one common
recovery point
– Allows an image copy to be taken while the database or area is online
�� Reduces the complexityReduces the complexity in establishing a recovery point for a database
Improved Usability
Improved Availability
ReducedComplexity
© 2011 IBM Corporation
Database Quiesce…
� Creating a recovery point withoutwithout Database Quiesce…– Issue /DBR on each IMS to establish a recovery point
•Each IMS will record a DEALLOC time in the RECON•Database resource is closed and data sets are deallocated
– Check to make sure the database or area was successfully taken offline on each IMS– Issue /STA on each IMS to allow database to be used
•OPEN option on /STA command is required to allocate and open the data sets
Otherwise first access will open the data sets
Improved Usability
Improved Availability
ReducedComplexity
IMS
RECONIMS
IMS
© 2011 IBM Corporation
Database Quiesce…
� Creating a recovery point withwith Database Quiesce…– Issue UPDATE DB|AREA|DATAGRP START(QUIESCE) command
•Command is processed by one IMS (command master) in the IMSplex
Databases and data sets are not closed and deallocated
Command waits for active use of the resource to reach a commit point and then quiesces the resource
Coordinated across the IMSplex, OLDS are switched on each IMS by default
RECON updated by one IMS with common DEALLOC timestamp on behalf of all the IMS systems
» New allocations for FP areas done on local IMS
» For FF databases, a new ALLOC is created at first database update after releasing the quiesce
Command master communicates with IMS systems to make resource available again
Improved Usability
Improved Availability
ReducedComplexity
IMS
RECONIMS
IMS
© 2011 IBM Corporation
Agenda
� IMS Databases: overview
� Database Quiesce (IMS v11)
� Open Database (IMS v11)
� XML DB (IMS v9) and XQuery (IMS v10)
� Integration with other Tools
© 2011 IBM Corporation
IMS Open Database
� Solution Statement– Open new avenues to IMS
• Offer scalable, distributed, and high-speed local access to IMS database resources
� Value– Business growth
• Allow more flexibility in accessing IMS data to meet growth challenges
– Market positioning• Allow IMS databases to be processed as a standards-based data
server
� Key differentiators– Standards-based approach (Java Connector Architecture, JDBC, SQL, DRDA)– Solution packaged with IMS
� Enables new application design frameworks and patterns– JCA 1.5 (Java EE)– JDBC
Innovative ImprovedUsability
OpenStandards
© 2011 IBM Corporation
Integrating IMS DB in a composite application
1. Channel: Web User Interface
3. Master Data Management Services
4. Operational Data Store: IMS DB
2. Process Services
ACCOUNT
ACCTID
ACCTNAME
BALANCE
CONTRACTID
TERMS
PRODID
© 2011 IBM Corporation
IMSIMSIMS
IMS
Mainframe
IMSIMSIMS
IMS
Mainframe
IMS Open Database
J2EE
JDBC Universal drivers
DLI
TCP/IP
TCP/IP
J2EE
JDBC Universal drivers
DLIDirect
z/OS
Distributed (e.g.; LUW)
Innovative ImprovedUsability
OpenStandards
© 2011 IBM Corporation
Solution highlights – connectivity
� Type 4 drivers– Universal drivers support type 4 connectivity to IMS databases from TCP/IP
enabled platforms and runtimes• WebSphere Application Server
• Windows
• zLinux
• z/OS
• Standalone Java SE
� Type 2 drivers– Universal drivers will support type 2 connectivity to IMS databases from z/OS
runtimes• WebSphere Application Server for z/OS
• IMS Java dependent regions
• CICS
– All environments zAAP eligible
Innovative ImprovedUsability
OpenStandards
© 2011 IBM Corporation
Solution highlights – Universal drivers
� Three Universal drivers
–Universal DB resource adapter• J2EE deployment
• Standard Java SE deployment
–Universal JDBC driver• Standard Java SE deployment
–Universal DLI driver• Standard Java SE deployment
Innovative ImprovedUsability
OpenStandards
© 2011 IBM Corporation
Solution highlights – J2EE deployment
� Universal DB resource adapter
–JCA 1.5• XA transaction support
Manage multiple datasource connections in a single UOW
• Local transaction support
Manage multiple datasource connections each in their own UOW
• Connection pooling
Pool released connections for future use
• Connection sharing
• Multiple programming models available
JDBC (Universal JDBC driver incorporated)
CCI with SQL interactions
CCI with DLI interactions
Innovative ImprovedUsability
OpenStandards
© 2011 IBM Corporation
Solution highlights – JDBC
� Universal JDBC driver– Significant enhancements to classic JDBC offered in IMS 9 and IMS 10
• Standardized SQL support
• XA transaction support (type 4)
• Local transaction support (type 4)
• Concurrency control
Control release of distributed locks
• Updatable result set support
• Batching support
Fetch multiple rows in a single network call
• JDBC metadata discovery support
� Standard SQL and metadata discovery enables significant integration opportunities for IMS
Innovative ImprovedUsability
OpenStandards
© 2011 IBM Corporation
Solution highlights – DLI
� Universal DLI driver
–Java implementation of DL/I API
–Complete DL/I support for database access
–All IMS command codes supported
–Can mix usage of JDBC and DLI drivers in the same application• SQL cannot always express what DLI offers
Innovative ImprovedUsability
OpenStandards
© 2011 IBM Corporation
Solution highlights – Java dependent region deployment
� Java dependent region resource adapter– Java framework for applications operating in an IMS container
• Message queue processing
• Program switching
Deferred and immediate
• Transaction demarcation
• GSAM support
• Additional IMS call support necessary for IMS transactions
INQY
INIT
LOG
Etc
– Shipped with type 2 Universal drivers
© 2011 IBM Corporation
Open Database Environment
LPAR Az/OS
IMS DBCTL
IMS
PCDistributed
UniversalDB
Resource
Adapter
J
CA
1.5
T
C
P
I
P
WebSphere
S
O
A
P
Evolution
LPAR B
LPAR C
XCF
TCP/IP
Innovative ImprovedUsability
OpenStandards
IMS Connect
T
C
P
I
P
SCI
SCI
O
D
B
A
ODBM
© 2011 IBM Corporation
Open Database EnvironmentInnovative Improved
UsabilityOpen
Standards
Universal
DB Resource
Adapter
J
C
A
1.5
T
C
P
I
P
WAS on Any Platform
S
O
A
P
LPAR Az/OS
Distributed
LPAR C
IMS DBCTL
IMS
PCSCI
O
D
B
A
IMS 11 Solutions
DLI
TCP/IP
JDBCTCP/IP
RYO DRDA Appl.
IMS DBCTL
IMS
PCSCI
O
D
B
A
IMS Universal Drivers
LPAR A
IMS DBCTL
IMS
PCSCI
O
D
B
A
ODBM
IMS Connect
T
C
P
I
P
SCI
ODBM
ODBM
LPAR B
XCF
z/OS WAS Universal
DB
Resource
Adapter
J
C
A
1.5
T
C
P
I
P
OD
BA
PCSCI
Traditional ODBA
DLI
JDBC
IMS Universal DriversTraditional ODBA
JEE
Non-JEE
© 2011 IBM Corporation
IMS Open Database – Performance Considerations
The JDBC and SQL
layer only adds
approximately 5%
overhead
4000 TPS
vs.
3800 TPS
Same language,
Same platform,
Different interface
C) DIST JAVA DLI
vs.
DIST JAVA SQL
System Z was
approximately 2 times
faster
4000 TPS
vs.
7892 TPS
Same call,
Same language,
Different platform
B) DIST JAVA DLI
vs.
SysZ JAVA DLI
Both COBOL & Java
meets business
requirements.
7892 TPS
vs.
9980 TPS
Same call,
Same platform,
Different language
A) SysZ Java DLI
vs.
SysZ COBOL DLI
Notes IBM SVL LabDescriptionScenario
© 2011 IBM Corporation
Agenda
� IMS Databases: overview
� Database Quiesce (IMS v11)
� Open Database (IMS v11)
� XML DB (IMS v9) and XQuery (IMS v10)
� Integration with other Tools
© 2011 IBM Corporation
IMS XML Database
� View/map native IMS hierarchical data as XML documents
� Allows the retrieval and storage of IMS Records as XML documents with no
change to existing IMS databases
� XML Evolution over XML Revolution
– Run XML transactions concurrently with traditional workload
XML Documents
IMS
Data
book
@year
title
seq
pricepublisherchoice
author
last first
seq
editor
last first
seq
affiliation
xs:date
xs:string
xs:string xs:string xs:string xs:string xs:string
xs:string xs:decimal
XML Schema
TITLE PUBLISH
FIRSTLAST FIRST
0:oo0:oo
AUTH EDIT
BOOK
YEAR PRICE
LAST AFFIL
PCB: BIB21
IMS DBD
© 2011 IBM Corporation
IMS XML-DB Metadata
� “Natural” mapping between hierarchic XML data and hierarchic IMS
database definitions.
PSB
DBD
IMS DB definition
<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:ims="http://www.ibm.com/ims"
xmlns="http://www.ibm.com/ims/PSBName/PCBName"
targetNamespace="http://www.ibm.com/ims/PSBName/PCBName"
elementFormDefault="qualified">
<xsd:annotation>
<xsd:appinfo>
<ims:DLI mode="store" PSB="AUTPSB11" PCB="AUTOLPCB"
dsg="DATASETG" meanLength="1000" numDocs="100"/>
</xsd:appinfo>
</xsd:annotation>
<xsd:element name=“A”>
<xsd:complexType>
<xsd:sequence>
<xsd:element name=“field1" type="xsd:int"/>
<xsd:element name=“field2">
<xsd:simpleType>
<xsd:restriction base="xsd:string">
<xsd:maxLength value="30"/>
</xsd:restriction>
…
XML Schema
mapping
XML view of IMS
data
XML document definition
© 2011 IBM Corporation
Agenda
� IMS Databases: overview
� Database Quiesce (IMS v11)
� Open Database (IMS v11)
� XML DB (IMS v9)
� Integration
© 2011 IBM Corporation
Integration
� IBM Mashup Center
– Web 2.0, Data as a service
� Data Source Explorer– IMS connectivity via Generic JDBC
– Database exploration
– View/Edit data
� Data Project Explorer– SQL scripting
• Graphical
• Content assist
� Cognos– Operational business intelligence and reporting
– IMS Connectivity via Generic JDBC
Offered by the
IMS Explorer
© 2011 IBM Corporation
IBM Mashup Center - Data as a service
� Web 2.0 support for IMS data
– Develop an IMS RESTful service/feed
• Generate meta files
from IMS DB
• IMS Web 2.0 Editor in InfoSphere MashupHub
© 2011 IBM Corporation
Web 2.0 support for IMS Data (Mashup Center)
� Enterprise Database Plugin– Allows users to create feeds from
any Database that supports a JDBC driver
� Supports the following databases:– DB2 – Derby– IDS– IMS
– Microsoft SQL– MySQL– Oracle
© 2011 IBM Corporation
Web 2.0 support for IMS Data (Mashup Center)
� Features:
– Support both managed and
unmanaged connections
– Can store connection profile
information for reuse
© 2011 IBM Corporation
Web 2.0 support for IMS Data (Mashup Center)
� Supports generating SQL queries via a GUI interface
© 2011 IBM Corporation
Operational Business Intelligence and reporting
� Type of BI that helps drive and optimize business operations on a daily basis
– Extends the use of BI to a much wider user audience
– Enables more timely business decisions; in relation to real time or right-time BI
Processing
– Allows business users to report on, analyze and optimize business operations
– Reduces the time between the discovery of problems or opportunities and taking action
on them
� Main pillars of BI are querying, reporting, OLAP, and business analytics
© 2011 IBM Corporation
The Three Levels of Business Intelligence
© Claudia Imhoff
© 2011 IBM Corporation
Reporting and OLAP ■ Enterprise Reporting
• Supports multiple report types: Production, Managed, Ad-hoc, Financial, etc
• Operates from a single metadata layer
• Can be personalized and targeted
• Can be distributed via email, portal, MS-Office, search application and mobile device
• Can assist in the management and optimization of daily business operations
• Capable of reporting on real-time data
• Presents complex data in a business way so it is easy to understand
■ Analysis
• Enables the guided exploration of information that pertains to all dimensions of your business
• Moves from summary level to detail levels of information effortlessly
• Analyzes and reports against online analytical processing (OLAP) and dimensionally aware relational sources
• Gets to the “why” behind an event or action to improve business performance.
• Identifies potential problems and recognizes business trends
© 2011 IBM Corporation
Cognos and IMS
� Ability to leverage IMS assets within the Cognos product family can be a tremendous
competitive advantage for any organization
� IMS 11 Open Database can enable this integration
– Removes inhibitors
• Prior to IMS 11, no built-in ad hoc query capabilities in IMS
• Standard tools often did not support hierarchical data; left to write own interfaces and
reports or had to purchase bridging tools
• Must be skilled in IMS to integrate with IMS and write reports
© 2011 IBM Corporation
IMS Value
� IMS is Efficient: 22K trans/sec on a single eight way z9. Remarkable performance that translates to the most cost efficient run-time environment. Reusing IMS transactions and data saves money.
� IMS enables Growth: IMS SOA provides Open, standard interfaces allowing ‘any-to-any’connectivity and access for new applications.
� IMS supports Growth: with IMS scalability you can just keep on growing. Customers have routinely handled peaks of 100 million transactions in a day.
� IMS is Resilient: Customers report going thousands of days without an unplanned outage. One bank reports availability of 99.998% over 10 years, 1.5 hours for planned outages.
� IMS and z/OS are secure
� Most efficient run-time, maintains application investment protection
� Fully integrated into today’s AD toolsets
� Fully integrates with WebSphere Application Server/ ESB / Process Server
� Fully supports SOA
� Improved efficiency and reduced skill requirements for systems and development staff
� Flexibility: JAVA, COBOL, C, PL/I, XML, Web 2.0…
� Continuous improvement to Rock solid availability
� Continuous improvement to operations and serviceability tools
� Fully supported by IMS & Tivoli tools
ResilienceResilience
GrowthGrowth
EfficiencyEfficiency
IMS “Value Proposition” 40+ years of Continuous Core Systems ImprovementIMS “Value Proposition” 40+ years of Continuous Core Systems Improvement
© 2011 IBM Corporation
The Message
� IMS continues to be a premier server with architected standard interfaces
– New products and tools from a variety of vendors provide access to IMS transactions
and data
� SOA is revolutionizing the way businesses are being designed and run. For it to make
sense:
– All assets must be easily accessible in a standard way
– All data must be represented and manipulated in a standard way
� Our goal is to leverage IMS Database Manager as an integral part of the enterprise in the
evolving business world through
– Addition of support for complimentary standards surrounding IMS data connectivity,
data representation, and application development
� And to allow you to realize the promises of building a Service Oriented Architecture:
– Simplify the business environment
– Respond to market changes more quickly and cheaply
© 2011 IBM Corporation