@LiaisonTech

Why Bad Data May Be Your Best Opportunity

@LiaisonTech

Bad Data Is…?

Not just…..

• Mal-formed file which won’t parse

• A document which tells lies

• Database of inconsistent information

• Out-of-date documentation

• An office rumor

• Commonly-accepted explanation which does not explain well

@LiaisonTech

What is Data?

“Data” for Our Purposes Is:

• All application or other stored data items

• All data items flowing through your enterprise

• All metadata for stored or flowing data items

@LiaisonTech

What is Bad Data?

“Bad” Data for Our Purposes Is:

• Difficult to understand

• Difficult to change

• Difficult to analyze

• Difficult to govern

@LiaisonTech

Consequences of Bad Data

• Dislike, Loathing

• Smells, looks, feels, sounds unpleasant

• Workarounds, Hacks

• Wastes money and limits business

Bad Data Has Consequences:

@LiaisonTech

Why Should You Care?

Today we will:

• Look at concrete, practical ways to begin taming bad data

• Learn from the mistakes of others

• Explore ideas so you can convince your boss your organization can do better

@LiaisonTech

Why Should You Care?

If you want to build a ship, don’t drum up people to collect wood and don’t

assign them tasks and work, but rather teach them to long for the endless immensity of

the sea.

- Antoine de Saint Exupéry*

* As quoted in A Mathematician’s Lament by Paul Lockhart

@LiaisonTech

Start by Exploring the Sea of Ideas

• Based on concrete, real-world challenges

• Expose patterns of bad and good data

• Lead to practical ways to get better

@LiaisonTech

Consistent, Offshore, Model-based SOA

Service WSDLS embed XML Schemas

First Consumer.

Usage of the operation schemas defined for that consumer.

@LiaisonTech

The Nth Consumer’s Needs & the Model

What is the correct way

to get from here

to here?

@LiaisonTech

The Nth Consumer’s Needs

• How to satisfy the next consumers’ needs consistently

- E.g. spreadsheets make consistency hard

• Micro use cases

• Communicating and enforcing the usage specification for a sprawling XML Schema

- XML Schema not designed for this

- Else unintended coupling among consumes over time

• Minor revisions of the service

• What tooling is needed?

@LiaisonTech

Offshore Development and SOA

• Message transformation between applications and the SOA service

• Offshore transformation development

• Time zone and cultural differences

- Accurate, unambiguous, complete specifications

• Governance to ensure only expected parts of messages populated

• What tooling is needed?

@LiaisonTech

Good Integration using SOA

• All consumers represent the same information in the say way in the service schema

• Consistent and accurate published specifications

• Infrastructure changes verified and governed

@LiaisonTech

Challenge: Rate of Change

• Point to point data flows are isolated

• SOA can mean service version changes, more rapid rate of change• Pub/Sub can have the same issues

• Greater need for accurate, managed, published specifications

• Greater need for governance

@LiaisonTech

Rate of Change

• Some customers change SOA XML Schemas twice a week during application development

• Forces software vendors to design for rapid rates of change

• Forces greater automation for creating and managing data integrations

@LiaisonTech

Automation Options

1. Derive specifications and implementations from the same artifacts

• Changing one changes the other

• SOA service specifications and data transformations from the same tool

• Linkages between tools, such as including enterprise vocabulary information in specifications

@LiaisonTech

Automation Options

2. Simplify User Experience

• Reduce cognitive load

- Guide data mapping based on specifications

• Facilitate communication

- Provide a common culture

- Capture, show, interpret information unambiguously

• Automate time-consuming tasks

- Tools not solutions

- “rather teach them to long for the endless immensity of the sea”

@LiaisonTech

Automation Options

3. Governance and Reuse

• Automate process steps

- Verify transformation steps against specifications

• Interoperability

- Integration with other tools

• Portability

- Runtime platform independence

@LiaisonTech

Automation Options

4. Low Barrier to Entry

• When start with simple steps, data captured for powerful results later

- Data transformations contain much information which can be reused later

• Easy integration with other tools

- No big porting efforts or reimplementation

@LiaisonTech

Liaison Contivo

Yes, this is part of what Contivo does. We designed it with this vision in mind.

@LiaisonTech

Bad to Good

• “SOA is pie-in-the-sky.” (pharma)

• “We maximized the cost of change.” (finance)

• “If he dies, nobody will….” (pharma)

• “If they can get it wrong, they will get it wrong.” (finance)

@LiaisonTech

Bad to Good Manifesto

• There is inherent order in your data

• Tools should extend the human

• Collaborative tools communicate corporate culture

• Invest in tools which let you start small and yield compounded returns

@LiaisonTech

Bad to Good Roadmap

1. Start doing your data integration projects with Contivo

• Seriously

• For many reasons

• Benefit from the experience of many others

• Define data transformations with Contivo Analyst

• It will capture much of what you will need to unsnarl the interwoven web of data you have now

@LiaisonTech

Bad to Good Roadmap

2. Do more data integrations with Contivo

• Store your maps, data samples, etc. in the Contivo Repository

@LiaisonTech

Bad To Good Roadmap

3. You will be able to

• See a big picture of your data

• Analyze your system of data integration requirements and options

• Postulate good data architecture directions

• Start doing projects in a way which will lead to good architectural results

• Have the data to justify and plan architectural overhauls

@LiaisonTech

Bad to Good Roadmap

4. Talk to us about your unique challenges

If you want to build a ship, don’t drum up people to collect wood and don’t assign them tasks and work, but rather teach them to long

for the endless immensity of the sea.

@LiaisonTech

The Order in Your Data

There is order in your data. Our job is to provide the tools so you can find and use it.

@LiaisonTech

Solutions

• Data Integration• Data Management • Data Security

Multinational

• Global headquarters in Atlanta• European offices in Finland,

Netherlands, Sweden, UK• More than 7000 customers

worldwide in over 46 countries

For more presentations:

Liaison Webinars

AB

OU

T L

IAIS

ON

28

Additional Resources