® ibm software group © ibm corporation linked data as an application integration architecture...

®

IBM Software Group

© IBM Corporation

Linked Data as an Application Integration Architecture

Martin Nally, IBM Fellow, Rational CTO

IBM Software Group | Rational software

2

My story

1. A case study – how we use linked data as an integration architecture

2. The gaps and problems we found, with proposed solutions

3. A call to action


3

Why do we need a new integration architecture?

Case study - IBM Rational in 2004 A leading provider of Application Lifecycle Management Tools

Source Code Management (ClearCase, #1 market share)

Defect/Change Management (ClearQuest, #1 market share)

Requirements Management (ReqPro, #2 market share).

…

Mature products

Designed and built in the ’90s

Evolving to the web


4

Times change – requirements change

New Requirements Distribution of development teams around the world

Want more tightly integrated end-to-end ALM solution

Want broader solutions

Need greater agility and lower cost of ownership

Response Adapt existing products to the new environment

Write glue code to integrate products

Necessary, works, but has limitations


5

Traditional Tool Integration. Ouch.

N2 possible point-to-point connectionsLimited coverage

Closed APIsVendor lock-in

Tight CouplingDependence on API details

Lockstep upgradesVersion incompatibilities

No place for cross tool functionsGlobal Query

Account MgmtTime for a new architecture …


6

What integration functions are really needed?

Create a link between ‘artifacts’ in different tools E.g Test links to requirement. Development task links to requirement

Within one tool, create an “artifact’ in another E.g. Within a Test tool, log a defect using test results

Across tools, share common concepts People form a team to work on a project to produce the next release of a product

Be able to query across information in multiple tools, e.g. Show me all the critical requirements that don’t have test cases

Show me al the new test cases I should run on last night’s build


7

Alternative Architecture - 1

Bus Proc Model

Software & Solution

ArchitectureDevelopmentEnt Arch

Require-ments

Test

Repository


8


Standard Application Server Architecture – tried and proven IMS, CICS, JEE, Tuxedo, Encina, …

Tried for tools many times before AD/Cycle, PCTE, …

Pros: Effective when the function can be delivered and deployed in a highly-

coordinated fashion

Cons: Requires schema coordination across tools

Does not provide a mechanism for integrating “existing ” or “foreign” tools

Requires “central planning” by consumer (deploy 1 database) as well as provider (integrate tools to a single schema)

Does not support an open ecosystem well


9


Bus Proc Model

Software & Solution

Architecture

DevelopmentEnt Arch

Require-ments

Test

“ESB – Enterprise Service Bus”


10


Pros: Allows integration of existing applications

Some of the process implemented in the “bus”

Cons: Multiple copies of data in different tools

Sensitive to changes in data model in any participating tool

Sensitive to changes in the process

Requires a lot of centralized implementation

Many of the disadvantages of traditional approachCentral point of control is bottleneck


11

Problem

We’ve been stuck with these problems and solutions for 20 years

Is Martin going to claim he’s invented a new way to write tools that fixes this? Yes, and No

We think there is another way

We didn’t invent it …

…Al Gore did


12

Linked Lifecycle DataData Integration for the 21st Century

Inspired by Internet principles, implemented with Internet technologies: simple interfaces for exchange of resources

Loosely coupled: everything is a “resource” linked together with URLs

Technology neutral: treats all implementations equally

Minimalist: defines no more than necessary for exchange of resources

Incremental: deliver value now, add more value over time

Openly published standards: free to implement and irrevocable

If the entire Web can connect like this, would the same idea work for ALM?


13


Pros: Allows integration of existing applications

Supports open, federated data model

Does not require data copying

Supports independent evolution of products

Cons:

Unproven (for our domain, at least)

Big paradigm shift (for us)

Lots of invention required

The rest of this talk focuses on our experiences


14

Identify Scenarios

Iterate on working drafts

Call it a spec

Gain technical consensus, collect non-

assert statements

Minimalist/additive approach Not a “complete” definition for a given area

Scenario driven scope

Co-evolve spec and implementations

Open participation around active core group

Open Services for Lifecycle Collaborationopen community. open interfaces. open possibilities

OSLC and Open Communityhttp://open-services.org


15

OSLC Community Eleven workgroups operating at open-

services.netDomain focused workgroups (e.g. CM,

QM, RM)Common issues and patterns (Core)Solution oriented workgroups (e.g.

PLM/ALM) Range of interests, expertise,

involvement400+ registered community members (up

from 70 people at RSC 2009) Individuals from 34+ different companies

have participated in OSLC workgroups (up from 5 companies at RSC 2009)

Worthy of notePLM/ALM Workgroup

Siemens leadershipOpen source

Eclipse Lyo Project Proposal Eclipse Mylyn project restructuring and

positioning of OSLC Mantis, Forges

Customer and integrator involvement GM, Northrop Grumman, Tieto, Integrate

Systems

AccentureAPGBlack DuckBoeingBSD GroupCitigroupEADSEmphasys GroupEmpulsysFokus FraunhoferGalorathGeneral MotorsHealth Care Services CorpIBMInstitut TELECOMIntegrate Systems

Lender Processing ServicesNorthrop GrummanOracleQSMRally SoftwareRavenflowShellSiemensSogetiSourceGear/TeampriseState StreetTasktop (Eclipse Mylyn)ThalesTietoTOPIC Embedded SystemsUrbanCodeWebLayers

15


16

Problem 0 - culture

People have been brainwashed for years XML (don’t get me started)

Operations, parameters

SPARQL Protocol for RDF is a typical example of the mess

Relational schemas and Object-oriented class models

Let’s invent new media types

Cling to closed world assumptions like to a security blanket

Desktop user interfaces


17

Problem 1 – creating data on the web

It seems obvious that you use POST to create, but what do you POST to?

How do I find the things that already exist?


18

Proposed Solution 1 – “Atom Publishing for LD”

A resource to which you POST to create new things, GET to find existing things Not really like APP, much simpler

But APP saw the need, defined the use-cases, got traction

We did not get this part right at http://open-services.net OSLC

Observation – the simplest way to define a collection in RDF is a set of triples with common subject and common predicate That is all there is to it!


19

Solution 1 - APP for LD example (Trig format)

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>.@prefix bp: <open-serives.org/bp/>. @prefix : <http://example.org/test/>.

:testCases = {

:testcases bp:commonSubject :proj1TestCases; bp:commonPredicate rdfs:member.

:proj1TestCases rdfs:member :testCase1; rdfs:member :testCase2. }

POST to this resource to create a resource -will also add a triple that points to the new resource

GET to find [triples that reference] existing resources


20

Problem 2 - But it will get big; I need to paginate

Yes, you need pagination APP has this

Solution proposal: Define <URL>?firstPage as a resource

Triples are a strict subset of those in URL (no change of subject)

Define :nextPage as a predicate


21

Solution 2 - Pagination example


:testCases?firstPage = {

:proj1Testcases rdfs:member :testCase1; rdfs:member :testCase2.

:testCases?firstPage bp:nextPage :testCases?page2 }.

:testCases?page2 = {

:proj1Testcases rdfs:member :testCase3; rdfs:member :testCase4.

:testCases?page2 bp:nextPage rdf:nil }.


22

Problem 3 - I only want the metaData

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>.@prefix bp: <open-serives.org/bp/> @prefix : <http://example.org/test/>.

:testCases?metaData = {

:testcases bp:commonSubject :proj1TestCases; bp:commonPredicate rdfs:member.

}


23

Basic Profile for LD

Problem 2 - Now know where to put them and find them - what does a resource look like?

1. Use URIs as names for things.

2. Use HTTP URIs so that people can look up those names.

3. When someone looks up a URI, provide useful information, using the standards (RDF*, SPARQL).

4. Include links to other URIs so that people can discover more things.

This is a terrific foundation, but … In practice, people have difficult interpreting this It doesn’t go far enough into detail for our purposes

Proposed Solution 2 – the “Basic Profile for LD” Mostly just some elaboration on the 4 rules


24

Basic Profile – 12 rules (and counting)

1. BP Resources are HTTP resources that can be created, modified, deleted and read using standard HTTP methods.

2. BP Resources use RDF to define their states.

1. POST any resource to a BP collection, not just a BP resource

2. Use the null relative URI for the subject when POSTing

3. You can request an RDF/XML representation of any BP Resource.

1. Turtle, please. JSON, please.

4. BP clients use Optimistic Collision Detection during update.

5. BP Resources use standard media types.


25


6. BP Resources use standard vocabularies.

Prefix Namespace URI

dc http://purl.org/dc/terms/

rdf http://www.w3.org/1999/02/22-rdf-syntax-ns#

rdfs http://www.w3.org/2000/01/rdf-schema#

bp http://open-services.net/ns/basicProfile#

xsd http://www.w3.org/2001/XMLSchema#


26


7. BP Resources use a restricted set of data types. Boolean: A Boolean type as specified by XSD Boolean

http://www.w3.org/2001/XMLSchema#boolean, reference: XSD Datatypes.

DateTime: A Date and Time type as specified by XSD dateTime http://www.w3.org/2001/XMLSchema#dateTime, reference: XSD Datatypes

Decimal: A decimal number type as specified by XSD Decimal http://www.w3.org/2001/XMLSchema#decimal, reference: XSD Datatypes

Double: A double floating point number type as specified by XSD Double http://www.w3.org/2001/XMLSchema#double, reference: XSD Datatypes.

Float: A floating point number type as specified by XSD Float http://www.w3.org/2001/XMLSchema#float, reference: XSD Datatypes.

Integer: An integer number type as specified by XSD Integer http://www.w3.org/2001/XMLSchema#integer, reference: XSD Datatypes.

String: A string type as specified by XSD String http://www.w3.org/2001/XMLSchema#string, reference: XSD Datatypes.

XMLLiteral: A literal XML value http://www.w3.org/1999/02/22-rdf-syntax-ns#XMLLiteral


27


8. BP Resources set rdf:type explicitly.

9. BP clients expect to encounter unknown properties and content.

10.BP clients do not assume the type of a resource at the end of a link.

11.BP servers implement simple validations for Create and Update.

12.BP Resources use simple RDF predicates to represent links.

1. “Anchors” in addition, not instead

1. only if adding properties to links

2. Use reified RDF statements for anchors


28

Problem 4 - Shapes (Validation, constraints)

RDFS and OWL do inferencing “If it walks like a duck, and quacks like a duck, it must be a duck”

In software (as in real life) it is also useful to go the other way “If it wants to be a good duck, it must walk like one and quack like one”

Everyone in software knows this model Database Schemas

Object-oriented programming class definitions

This model is actually very important for writing practical software

Need to support more than one perspective on a duck It’s a food-source, a source of insulating materials, a hazard to aircraft, a bird

Problem - this is a gap in RDF technologies


29

Proposed Solution 4 - Shapes


:teatCaseShape = { :testCaseShape a bp:Shape; bp:describedClass :TestCase; pb:propertyConstraint :testsRequirementConstraint }.

:testsRequirementConstraint = { :testsRequirementConstraint a bp:propertyConstraint; bp:describedProperty :testsRequirement; bp:rangeShape :requirementShape }.


30

Problem 4 - Updating data on the web

You use PUT, right?

Observation – SQL has no equivalent of PUT

Problem 4 – PUT is good for documents, not a good idea for data Read-only properties

Independent perspectives

Proposed solution 4 “Put” more energy into PATCH


31

Cross-product query

Query server independent of storage, broader scope

Schema-less query

Exploit RDF XML and XQuery were tried, did not work as well


32

Implementing query on data not in a repository

HTTP GET/PUT/POST/DELETEHTTP GET/PUT/POST/DELETE

JenaRDF represent’n

RDF represent’n

SPARQL


33

UI Integration

Problem 5: A test product wants to allow testers to open defects in another product

Defects are customized – don’t know in advance what fields they have

Defects have context (projects, components) and “meta-data” (defect states, defect types, …)

Option 1 – teach test tool about defects, their types, constraints, contexts

Option 2 – let the test tool use a UI component from the defect tool to create the defect allows much simpler interface, looser coupling

Relies on human third-party

Problem 6 – “hover help” on links

Solution – UI preview protocol


34

Security

Requirements Allow each product to have its own security administration and

authentication

Provide optional central place to configure users, integrate with LDAP etc

Allow each product to manage it’s own access control lists, pre-conditions and post-actions

Provide optional central federation of ACL administration

Solutions For products that implement their own authentication, federate with OAuth

For products that want to manage ACLs, provide federation

Provide optional building blocks for new-construction that take over admin, authentication and ACLs


35

How do you write an Ajax UI?

Attempt 1 – AJAX clients handle multiple-resources Pros:

Responsive UI – no page refresh between resources

Cons:

Complex clients (1990s client-server)

Closed system (can’t link to unexpected resources)

Future attempt: Page-per-resource

Ajax inside pages


36

Miscellaneous architecture/design

Tags, not folders (still some controversy)

Relationships/predicates, not collections (still some controversy)

Exploit RDF, eschew WebDAV

Programmable resources (but stay RESTful)

Back links for inverse navigation?

Discovery

What is userID?

Transactions


37

Old NewServices REST

Files Resources

Repositories Linked open data

J2EE apps Heterogeneous apps

Tightly-coupled apps Loosely-coupled apps

Relational/XML data/Objects RDF

SQL/XQuery SPARQL

authentication OAuth

Folders Tags

WebDAV RDF

Collections RDF Predicates

Need more help to shift to a web-centric design


38

Conclusion

Linked Data could be huge in enterprise computing a very significant new application integration model

Some significant gaps to close if we are to succeedBetter definition and documentation of best practices, e.g.

12 practices of Basic Profile

UI guidelines (resource-oriented UI)

A few important specification gaps, e.g.

POST-able collections

Shapes/constraints/validation

Security

Let the W3C take up the challenge!

® ibm software group © ibm corporation linked data as an application integration architecture...

Documents