Data Integration in Service Oriented Data Integration in Service Oriented ArchitecturesArchitectures

Rahul PatelRahul PatelSr. Director R & D, BEA Systems

Liquid Data – XML-based data access and integration for enterprise data

Project Alchemy – Mobile, disconnected applications for enterprise data

June 8, 2004

Service Oriented ArchitectureService Oriented Architecture

Employees Customers Partners

Specialty InfrastructureDirectory, Search, Content, etc

Enterprise ApplicationsSFA, CRM, Custom, Legacy, Mainframe

Database & Tuxedo

Data & Transaction Services Infrastructure Services

Message Bus (Transport, Routing, Transformation)

Shared Business Services

1

Composite Applications, Portals and B2B Services

Man

agem

ent

Sec

uri

ty

Enterprise Data Services Enterprise Data Services

Customer

(DB)

Order Mgt

(DB)

Customer

Portal

GetCust_by_Custid

GetCust _by_Orderid

Update_Address

Create_Order

Customer:Jack

Customer

(Package

App)

Order Mgt

(Web

Service)

Division 1

Division 2

Requires accessing and integrating data from multiple resources

Service Clients

Data Services on Logical Model Physical Model

AddressesCustomer

Profile

Pending Payments

Credited Payments

Orders

History

Data Services PlatformData Services Platform

Repository

Enterprise Data

JDBC

Model DrivenIntegration(XQuery)

Customer

Order

Payment

ProgrammingModel(SDO)

Web Srvcs

Adapters

Custom

J2E

E,

We

b S

erv

ice

, X

Qu

ery

, J

MS

•Logical models capture the data access & integration complexity ONCE•SAME data model, programming model, & API for whole enterprise

Enterprise Data ServicesEnterprise Data ServicesModel Based ApproachModel Based Approach

Model-based unification of disparate data sources– Define virtual model(s)– Map physical data source models to virtual model– Data Mediator converts operations against virtual model

intoQueries and transactions against individual sources,

plusMiddleware coordination and computations to finish the

job Technology for Enterprise Data Services

– Data model mapping and integration– SOA-appropriate programming model– Translation of data models, translation of queries– Performance and Caching

Choosing Appropriate TechnologiesChoosing Appropriate TechnologiesXML is the AnswerXML is the Answer

Required Capabilities Standards

XMLA standard for data format and data interchange

A standard for querying both relational & non-relational data

XQuery

A standard for interfacing into applications

Web Services, JCA

XML SchemaA standard for describingand modeling data

A standard for publishing available services

Web Services

Data Services InfrastructureData Services InfrastructureMust tie all the pieces togetherMust tie all the pieces together

Various parts of data architecture– XMLSchema (shape)– Logical Views (nodes in the information model)– Relationships (arcs in the information model)– Data sources (enterprise data source)– XQuery functions/source (aggregate/transform logic)– Programming model for these (SDO)– Business logic (Java, Workflows, etc.)– Publishing the models for application use (Web Services, SDO,

Controls..)– Policies (meta-data, security, etc.)

How to connect all these in a meta-data environment– Dependencies– Integrated development / management experience– A logical of reusable unit of data representing a particular business

entity – e.g. Single view of Customer

Model Driven “Read” Data ServicesModel Driven “Read” Data ServicesOptimized Distributed Queries on Logical ModelOptimized Distributed Queries on Logical Model

SQL sub-query pushdown to relational sources– Selects, projects, functions,

inner/outer joins, sorts, aggregates

Distributed join optimization– Batched parameter passing

Join, Index Join, Merge Join

Slow data source handling– Intra-query function result

caching– Parallel invocation of slow

functional sources

Query result cache for stored queries (optional)– Consistency based on TTL plus

invalidation API

Check security & cache

Cache results

Create optimized Execution plan

Merge & Transform Data

1

2 4

5

Data ServiceRequest

QueryResult

Get data from underlying sources

Sub-query Sub-query Function Call

3

Model Driven “Write” ServicesUpdate Logic specified on the same Model

Physical Source 1

Physical Source 2

Update_Address

Update_CreditCard

MappingMeta-data

MappingMeta-data

Decomposition

Change LogTime StampSource Info.Update Plan

User Defined Logic,Validation, Rules, etc.

User Defined Logic,Validation, Rules, etc.

User Exit on View

CachingCaching

Latency in SOA drives tremendous need for caching

Caching in middleware is inherently “optimistic”– Optimized for reads– Optimistic for writes

Caching granularity should at logical data views and navigations between them– Caching individual data source results is not

sufficient