middleware diarrhea and other ailments

1

Middleware Diarrhea and Other Ailments

Michael Stonebraker

Adjunct Professor

Massachusetts Institute of Technology

([email protected])

2

M.I.T.

OutlineOutlineOutlineOutline

Too much middlewareXML ailmentsWeb services illsOur professional sickness

Too much middlewareXML ailmentsWeb services illsOur professional sickness

3

M.I.T.

Client-Server Got Replaced by N-Tier Client-Server Got Replaced by N-Tier ComputingComputingClient-Server Got Replaced by N-Tier Client-Server Got Replaced by N-Tier ComputingComputing

The WebGizmosScalability and management problems

with client server

The WebGizmosScalability and management problems

with client server

4

M.I.T.

Humility LessonHumility LessonHumility LessonHumility Lesson

We all sold client-server hardduring the 80’sand even into the 90’s

Less than 10 years later it is the worst idea on the planet

We all sold client-server hardduring the 80’sand even into the 90’s

Less than 10 years later it is the worst idea on the planet

We should feel really dumb!

5

M.I.T.

N-Tier Computing Produced Lots of N-Tier Computing Produced Lots of MiddlewareMiddlewareN-Tier Computing Produced Lots of N-Tier Computing Produced Lots of MiddlewareMiddleware

App serversEAI/messagingETLFederatorsWorkflowCMSPortalsDBMS

App serversEAI/messagingETLFederatorsWorkflowCMSPortalsDBMS

6

M.I.T.

Middleware DiarrheaMiddleware DiarrheaMiddleware DiarrheaMiddleware Diarrhea

Average enterprise has one (or more) app serversone (or more) EAI packages one (or more) ETL packagesone (or more) portal productsone (or more) application packagesand maybe someday a federated

DBMS

Average enterprise has one (or more) app serversone (or more) EAI packages one (or more) ETL packagesone (or more) portal productsone (or more) application packagesand maybe someday a federated

DBMS

7

M.I.T.

All of these systemsAll of these systems All of these systemsAll of these systems

Contain transformation enginesAnd often do function activation (app service)And often have adapters to legacy systems

Contain transformation enginesAnd often do function activation (app service)And often have adapters to legacy systems

Huge overlap in functionality!!

8

M.I.T.

Less Moving PartsLess Moving PartsLess Moving PartsLess Moving Parts

Less systemsMore uniformityLess duplication

Less systemsMore uniformityLess duplication

9

M.I.T.

Less SystemsLess SystemsLess SystemsLess Systems

Less system administratorsLess trainingLess manualsLess bugsLess cross system issues

Less system administratorsLess trainingLess manualsLess bugsLess cross system issues

10

M.I.T.

More UniformityMore UniformityMore UniformityMore Uniformity

Every island has memory managementsecurity modelthreading model

Less is better

Every island has memory managementsecurity modelthreading model

Less is better

11

M.I.T.

Less DuplicationLess DuplicationLess DuplicationLess Duplication

Most of the islands support transformationsreasonable chance you will do each one 6

or more timesmaintenance headache

Most of the islands support transformationsreasonable chance you will do each one 6

or more timesmaintenance headache

12

M.I.T.

So How To Consolidate…… So How To Consolidate…… So How To Consolidate…… So How To Consolidate……

Converge app server into OR DBMSdumbest OR query is execute function

Converge app server into OR DBMSdumbest OR query is execute function

Remember that everything looks like a

nail to the guy with the hammer!

13

M.I.T.

PictoriallyPictorially

client

DBMS

DBMS

component

component

component

14

M.I.T.

This Requires….This Requires….This Requires….This Requires….

DBMS to send queries to other DBMSsI.e. be a data federatorLoad balance also requires a federator

DBMS to send queries to other DBMSsI.e. be a data federatorLoad balance also requires a federator

15

M.I.T.

Best of Breed FederatorsBest of Breed FederatorsBest of Breed FederatorsBest of Breed Federators

Support schema heterogeneityby executing OR functions

Support materialized viewsto cache static data

Support schema heterogeneityby executing OR functions

Support materialized viewsto cache static data

16

M.I.T.

Less Moving Parts….Less Moving Parts….Less Moving Parts….Less Moving Parts….

Federators dominate ETLETL only supports “push”federators do both “push” and “pull”

Federators dominate ETLETL only supports “push”federators do both “push” and “pull”

17

M.I.T.

WorkflowWorkflowWorkflowWorkflow

A collection of ruleswho’s allowed to buy whatand who must approve it

Best considered as a boxes and arrows

diagramAnd compiled into components to run on an

app server

A collection of ruleswho’s allowed to buy whatand who must approve it

Best considered as a boxes and arrows

diagramAnd compiled into components to run on an

app server

18

M.I.T.

Workflow Framework -- PO’s

managerPOPO Big? nono

Lavenyesyes

IT? nono

yesyes

19

M.I.T.

Data Intensive Workflow Should Move Inside an OR DBMSData Intensive Workflow Should Move Inside an OR DBMS

GUI for “boxes and arrows”Compiler for the diagram

processing steps become components business rules become triggers all data flow inside the DBMS

Worked great in Media/360

GUI for “boxes and arrows”Compiler for the diagram

processing steps become components business rules become triggers all data flow inside the DBMS

Worked great in Media/360

20

M.I.T.

Why?Why?

Big Big Big performance advantage no polling of the DBMS no data movement easy to change!

Big Big Big performance advantage no polling of the DBMS no data movement easy to change!

Watch for Informix product in this area!

21

M.I.T.

NirvanaNirvanaNirvanaNirvana

One integrated system that doesfederationEAI app service

With a single transformation systemBased on DBMS technology (or

something else….)

One integrated system that doesfederationEAI app service

With a single transformation systemBased on DBMS technology (or

something else….)

22

M.I.T.

XMLXMLXMLXML

Good for content storage and movementGood as “on the wire” format for data

movementas long as you don’t need to send a

lot of stuff fastBad for data storage!

Good for content storage and movementGood as “on the wire” format for data

movementas long as you don’t need to send a

lot of stuff fastBad for data storage!

23

M.I.T.

History LessonHistory LessonHistory LessonHistory Lesson

1960’s IMS and IDMS get tractioncustomers start complaining about

rewriting everything when schema

changes

1960’s IMS and IDMS get tractioncustomers start complaining about

rewriting everything when schema

changes

24

M.I.T.

History LessonHistory LessonHistory LessonHistory Lesson

1970Codd writes pioneering paperstarts a decade long argument between

IMS/CODASYL advocates and Codd

supporters

1970Codd writes pioneering paperstarts a decade long argument between

IMS/CODASYL advocates and Codd

supporters

25

M.I.T.

Net-Net of ArgumentNet-Net of ArgumentNet-Net of ArgumentNet-Net of Argument

Putting semantics into data order is badrestricts storage options

Hidden meaning bad no self-defining fields

Putting semantics into data order is badrestricts storage options

Hidden meaning bad no self-defining fields

26

M.I.T.


Data independence is goodschemas change oftendon’t want to rewrite anything when this

happens

Data independence is goodschemas change oftendon’t want to rewrite anything when this

happens

27

M.I.T.


Complexity is badhigh level query languages are goodKISS arguments

Call these three premises “Codd’s laws”

Complexity is badhigh level query languages are goodKISS arguments

Call these three premises “Codd’s laws”

28

M.I.T.

History LessonHistory Lesson History LessonHistory Lesson

1983(?)Codd wins Turing awardacknowledgement for being right

1983(?)Codd wins Turing awardacknowledgement for being right

29

M.I.T.

XML in This Historical LightXML in This Historical Light XML in This Historical LightXML in This Historical Light

Most of the bad features of IMS/Codasylallows semantics in data orderdata independence will be a challenge

try updates on inverted hierarchies look at IMS LDBs

more complex than Codasyl

Most of the bad features of IMS/Codasylallows semantics in data orderdata independence will be a challenge

try updates on inverted hierarchies look at IMS LDBs

more complex than Codasyl

30

M.I.T.

Our FieldOur Field Our FieldOur Field

We look a little silly sayingan idea renounced in the 1970’s is back

Leading our colleagues to ask “What’s

different?” if somebody disproved Codd’s laws; they

didn’t tell me…..

We look a little silly sayingan idea renounced in the 1970’s is back

Leading our colleagues to ask “What’s

different?” if somebody disproved Codd’s laws; they

didn’t tell me…..

31

M.I.T.

How to Win the Turing Award Circa 2020How to Win the Turing Award Circa 2020How to Win the Turing Award Circa 2020How to Win the Turing Award Circa 2020

2000’sXML data storage gets traction

2010dust off Codd’s paper

Wait 10 years to be proven right

2000’sXML data storage gets traction

2010dust off Codd’s paper

Wait 10 years to be proven right

32

M.I.T.

In Any CaseIn Any CaseIn Any CaseIn Any Case

In line tags turn 1Tbyte of EMP data into 10

Tbytes of EMP dataWon’t store anything big in native XML

will use something else…. like what?

In line tags turn 1Tbyte of EMP data into 10

Tbytes of EMP dataWon’t store anything big in native XML

will use something else…. like what?

33

M.I.T.

OR DBMSOR DBMS OR DBMSOR DBMS

XML is merely this year’s data typeNext year it will be WML or …

and there will be a next year….

XML is merely this year’s data typeNext year it will be WML or …

and there will be a next year….

34

M.I.T.

XMLSchemaXMLSchema XMLSchemaXMLSchema

Contains the kitchen sinkComplexity run amok

diarrhea from the SGML typesIncludes lots of known hard stuff

e.g. union types

Contains the kitchen sinkComplexity run amok

diarrhea from the SGML typesIncludes lots of known hard stuff

e.g. union types

35

M.I.T.

XqueryXquery XqueryXquery

Mostly syntactic sugar on OR SQL // is a user-defined function in

Informix OR engineTry to keep the semantics close to OR

SQL

Mostly syntactic sugar on OR SQL // is a user-defined function in

Informix OR engineTry to keep the semantics close to OR

SQL

36

M.I.T.

Another History LessonAnother History LessonAnother History LessonAnother History Lesson

Typical enterprise wanted data integration

for business analysis badlyneeded data in a variety of systems in a variety of formatsoften with no unique idsoften with incompatible semantics

2 day delivery means lots of thingsoften dirty

Typical enterprise wanted data integration

for business analysis badlyneeded data in a variety of systems in a variety of formatsoften with no unique idsoften with incompatible semantics

2 day delivery means lots of thingsoften dirty

37

M.I.T.

ETL Warehouse Projects of the 90’sETL Warehouse Projects of the 90’sETL Warehouse Projects of the 90’sETL Warehouse Projects of the 90’s

Well into 8 digitsUsually a factor of three behind scheduleDelivering a factor of 3 less stuffEverybody dented their pick on semantic

heterogeneitywhich is hard, hard, hardand not solved by the blizzard of 3 letter

acronyms from Redmond

Well into 8 digitsUsually a factor of three behind scheduleDelivering a factor of 3 less stuffEverybody dented their pick on semantic

heterogeneitywhich is hard, hard, hardand not solved by the blizzard of 3 letter

acronyms from Redmond

38

M.I.T.

Web ServicesWeb ServicesWeb ServicesWeb Services

Will be a long time coming outside of

simple domains (where there is no data

integration to deal with)E.g. catalog management

Grainger perspiration….

Will be a long time coming outside of

simple domains (where there is no data

integration to deal with)E.g. catalog management

Grainger perspiration….

39

M.I.T.

The Depressing State of AffairsThe Depressing State of Affairs

~50-75% of IT projects fail if we built bridges, our profession

would be firedand the same mistakes are repeated

over and over (excessive ambition,

rolling specs, bad design, failure to

load a large data set early)

~50-75% of IT projects fail if we built bridges, our profession

would be firedand the same mistakes are repeated

over and over (excessive ambition,

rolling specs, bad design, failure to

load a large data set early)

40

M.I.T.

What To Do?What To Do?

We typically don’t teach this stuff (and

do a serious disservice to our students)probably because we don’t (can’t)

spend any time in industry to figure it

out

We typically don’t teach this stuff (and

do a serious disservice to our students)probably because we don’t (can’t)

spend any time in industry to figure it

out

Action item: at the very least read a couple of Robert L. Glass’s books

41

M.I.T.

The Depressing State of AffairsThe Depressing State of Affairs

Hardware “half-life” is 18 monthsSoftware half-life is 18 years (or more)!In 25 years we moved from

C to JavaSQL to Xquery

Hardware “half-life” is 18 monthsSoftware half-life is 18 years (or more)!In 25 years we moved from

C to JavaSQL to Xquery

42

M.I.T.

What To Do?What To Do?

Much higher level design environmentsvisworkflowspecial purpose languages (report

writers,…)And stop turning down papers on this

stuff

Much higher level design environmentsvisworkflowspecial purpose languages (report

writers,…)And stop turning down papers on this

stuff

43

M.I.T.

Grand ChallengeGrand Challenge

Improve application productivity

(probability of success * programmer

productivity) by 2 this decade

Improve application productivity

(probability of success * programmer

productivity) by 2 this decade

middleware diarrhea and other ailments

Documents