infosphere: leading from the front - accelerating data integration through metadata

Leading from the FrontAccelerating Data Integration through MetadataScott AbbottCertified IT Architect, InfoSphere Software

Make change work for youIBM Insight Forum 09®

C t tContext


22IBM Insight Forum 09®

Make change work for you

Are you e youconstantly disappointeddisappointed by your Data I t tiIntegration projects?


Often it’s because we rush in without thinkingthinking what we are d idoing


Typical Data Integration Project REPORTSTypical Data Integration ProjectOLAP

4WAREHOUSE

DATA INTEGRATIONDATAMARTS

12 3

LEGACYSOURCES

REFERENCE DATA “if we build it they will come”

MASTER DATA

“The custom data model”

“of course our data is good”

“we’ll work it out in the testing”


Th I f S h S ft E l tiThe InfoSphere Software Evolution

Ch D tDataMirror

LAS Global Name

Change Data Capture

DWLOperational Master Data

Management

Global Name Enrichment

Unicorn

TrigoSRD

Ascential

Transformation, Cleansing, Profiling and metadata integration

Entity Resolution and

Metadata Management

Product Information Management

Entity Resolution and Analysis


InfoSphere Information Server



4WAREHOUSE


12 3

LEGACYSOURCES

REFERENCE DATA

MASTER DATA


METADATA

Pitf ll #1Pitfall #1

“Th C t M d l”“The Custom Model”




DI Pitfall #1WAREHOUSE

1

“The custom data model”

“ h k i d

data model

NZ Customer Experience

“who knows our industry better than us”

• Project duration 24-36 mths• Model never fully deployed• Complex ETL feeds d t bili d ti BI t“it will only take a couple of

months”

destabilized entire BI system• Users bypass to get required information


DI Pitfall #1 AcceleratorAccelerator

80:20 rule (20% customization)80:20 rule (20% customization) Months not years

Fully attributed data models across six industries

C l t b i t l t fComplete business templates for industry KPIs

Ke accelerators for migration &Key accelerators for migration & integration projects

A t l ti t l t ithiAct as acceleration templates within Information Server & Cognos 8 BI



4WAREHOUSE

industry


12 3

LEGACYSOURCES

industry models

REFERENCE DATA

MASTER DATA

Target state

Target state


METADATA


if b ild itif we build itthey will come..y




14DI Pitfall #2

OLAP

REPORTS

44

“if we build it they will come”

“it is what the business

they will come”


asked for” • Multiple examples of BI solutions not meeting initial business driversU i BI“the users will understand

the new system”• Users perceive new BI initiatives as burdens rather than assets


15Missing the PointC t Chi WhiCorporate Chinese Whispers

Identify High Value Customers to support

Call Centre & Web

Monthly Report on Customers Revenue

breakdownCall Centre & Web Personalization

breakdown

DBAsArchitectsSubject Matter Experts

Business Users

DevelopersDataAnalysts

IBM Insight Forum 09®


16Bridging the Gapl ti th t th ldrelating the new to the old

“item”

“component” “part”?

??




26


29

U d t di Y D tUnderstanding Your Data

InfoSphere Business Glossary

Captures Business TaxonomiesCaptures and defines shared searchable business glossaryAssigns stewardship to key business termsLinks business terms to technical assets


InfoSphere Business GlossaryInfoSphere Business GlossaryWeb-based authoring, managing and sharing of business metadataAligns the efforts of IT with the goals of the business Provides business context to

Subject Matter Experts

I f S h B i Gl

Business Users

information technology assetsEstablishes responsibility and accountability

Create and manage business vocabulary and relationships, while

linking to physical sources

InfoSphere Business Glossary

y linking to physical sources

GL Account Database = DB2Number

The ten digit account number. Sometimes referred to as th t ID

Schema = NAACCT

Table = DLYTRANS

C l Technical Business

Business View

the account ID. This value is of the form L-FIIIIVVVV.

Column = ACCT_NO

data type = char(11)

Technical


Business Glossary Anywhere ANYBusiness Glossary AnywhereReal-time access to business glossary from any desktop application

ANY User

FeaturesFrom any desktop application, click on a term & view its business definition in a pop-up window without any loss of context or focusI t lli t t hi t b t did t i

From Any Application..

.

Intelligent matching returns best candidates in a single searchSearch engine for terms and categoriesAccess steward contact information directlySecurity enforced via the Information Server common security layer

BenefitsIncreased trust and acceptance of information by delivering definitions in contextExpanded adoption of enterprise glossary outside ofExpanded adoption of enterprise glossary outside of Information Platform technologiesImproved information availability with multiple access mechanisms for electronically stored information (ESI)

Pop the Definition!


4WAREHOUSE


12 3

LEGACYSOURCES Correct

REFERENCE DATA

Data Steward

Data Steward

Understood

MASTER DATA

TermsTerms

Target state

Target state


METADATA


d t litdata quality




DI Pitfall #3

2

LEGACYSOURCES

2

“of course our data is good”

“ h b i h


“the business owner says the information we need is in there”

• ETL Proof of Concept• Client assured data quality sufficient so

excluded data cleansing from scope• At end of 2wk pilot, project halted due to

unsolvable data quality issues

“the schema’s show they have the same keys”

q y

• Many 15-20 year old systems still in operation in NZ market



38


39


40


41


42


43


44


45


46


47


48


49


50


51


52


53


54


55


56


57


58


59

InfoSphere Information AnalyzerInfoSphere Information Analyzer

Data-centric analysis of application, database and file-based sources Data

AnalystsSubject Matter

Experts

Secure, detailed profiling of fields, across fields, and across sources

Analyse source data structures, and monitor adherence to integration and

lit l

InfoSphere Information Analyzer

Creation of metadata from profiling results

Results instantly promotable across

quality rules

Results instantly promotable across IBM InfoSphere Information Server

Physical View



4WAREHOUSE


12 3

LEGACYSOURCES

Correct

REFERENCE DATA

Data Steward

Data Steward

Understood

MASTER DATA

TermsTerms

Target state

Target stateSource

StateSource State

ETLHints


METADATA


It tiIterative Developmentp




DI Pitfall #4

DATA INTEGRATION3

“we’ll work it out in the testing”


• ETL development >75% total project $$P j t t ki 2 3 l th l d• Projects taking 2-3x longer than planned

• Some clients taking 70+% of dev.time doing impact analysis• Impact analysis methods very basic• Largely iterative development method• Unreliable forecast completion dates• Low levels of trust by business in IT ability to achieve BI

outcomes• Substantial cost overruns• Expensive BI maintenance costs


H d I Fi d O tWhere does the

data for thisHow do I Find Out …Data Analyst

data for this report come

from?

…where this data comes from?

… when the job had been running last time?

… the details for these assets?




D l tDevelopment(Impact Analysis)( p y )





80

What is the InfoSphere Metadata Workbench?What is the InfoSphere Metadata Workbench? Web-based exploration of Information Assets generated and

Data I t ti Developers

gused by Information Server applicationsOut of the box reporting on data

Integration Managers

Developers

Provides IT professionals with a tool for

InfoSphere Metadata Workbench®

p gmovement, data lineage, business meaning, impact of changes and dependencies

Provides IT professionals with a tool for exploring and understanding the assets generated and used by the Information Server suite.

Tracing the data lineage of Business Intelligence Reports to provide basis for compliance with

Slegislation such as Sarbanes-Oxley and Basel II


4WAREHOUSE


12 3

LEGACYSOURCES

Correct

REFERENCE DATA

Data Steward

Data Steward

Understood

MASTER DATA

TermsTermsImpact AnalysisImpact

Analysis

Target state

Target stateSource

StateSource State

ETLHints


METADATA


D l tDevelopment(Iterative cycles)( y )





4WAREHOUSE


12 3

LEGACYSOURCES

Correct

REFERENCE DATA

Data Steward

Data Steward

UnderstoodRequirements

ETL Code GenerationETL Code

Generation

MASTER DATA


Analysis

Target state

Target stateSource

StateSource State

ETLHints


METADATA

InfoSphere FastTrack

Business analysts and IT

InfoSphere FastTrackTo reduce costs of integration projects through automation

Business analysts and IT collaborate in context to create project specification

Leverages source analysis

Specification

Leverages source analysis, target models, and metadata to facilitate mapping process

Auto-generation of data transformation jobs and reportsj p

Auto-generates DataStage jobs

Flexible Reporting


4WAREHOUSE


12 3

LEGACYSOURCES

Correct

REFERENCE DATA

Data Steward

Data Steward

UnderstoodRequirements

ETL Code GenerationETL Code

Generation

MASTER DATA


Analysis

Target state

Target stateSource

StateSource State

ETLHints


METADATA

93Information ServerO ti i i A li ti D l tOptimizing Application Development



IBM InfoSphere Information Server94

IBM InfoSphere Information ServerDelivering information you can trust

I f ti SInformation Server

Information Services DirectorInfoSphere

Data Architect

Information AnalyzerInfoSphere

Business GlossaryInfoSphereQualityStageInfoSphere DataStageInfoSphere

Federation ServerInfoSphere

Replication Server / EVPInfoSphereInfoSphere

FastTrackInfoSphere Change Data CaptureInfoSphere

Metadata ServerInfoSphere

Metadata WorkbenchInfoSphere Metadata WorkbenchInfoSphere


95Bringing It All Togetherg g g

DevelopersSubject Matter Experts

DataAnalysts

Business Users

Architects DBAs

Simplify Integration Increase trust and confidence in informationI li tF ilit t h

Information Server – Common Framework

Increase compliance to standards

Facilitate change management & reuseDesign Operational



Leading from the FrontGreater Preparation will yield dramatically lowerGreater Preparation will yield dramatically lower project costs/times

Typical Work Effort for Migration Activities

15-30% of total project budget will be spent on Migration Activities15-30% of total project budget will be spent on Migration Activities15 30% of total project budget will be spent on Migration Activitiesp j g p g

30%Understanding

40%Cleaning, Standardising

30%Conversion, Loading,

DeliverDiscover Prepare

Largely manual effort on small percentage of data. Some manual

This effort is the most unpredictable. The work can vary greatly depending on condition of data, however it is always the largest piece of work in the data initiative.

Largely manual effort on 100% of data. This can mean d f l i t ll t

Coding transformations and loads. Traditionally this effort is plagued with problems related to data quality and it

can easily be pulled by necessity into the

75% Business 50% Business 25% Business

Source Data Harmonizing, Management Interfaces, Connectivity

percentage of data. Some manual coding can review all data . dozens of persons cleaning source systems manually to

correct and augment data and manually aligning records to MRD. Some manual coding can reduce the manual

effort.

can easily be pulled by necessity into the Cleaning, Standardising and Harmonising

area causing timing and budget problems.

75% IT50% IT25% IT



97

Th kThank you

Questions?Questions?



infosphere: leading from the front - accelerating data integration through metadata

Technology

change work

bi ibm insight forum

y ibm insight forum

testing ibm insight

contextc t t ibm insight

d i ibm insight forum

business metadata

ai t lli t t