building a reusable data integration framework
DESCRIPTION
Building a Reusable Data Integration Framework. Strategic Agility. Matt Siano Principal Database Engineer HBS Information Technology June 2011. David Aznavoorian Director, Database Services HBS Information Technology June 2011. Agenda . Evolution of Enterprise Data Integration - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/1.jpg)
Copyright © President & Fellows of Harvard College
Building a Reusable Data Integration FrameworkStrategic Agility
David AznavoorianDirector, Database Services HBS Information TechnologyJune 2011
Matt SianoPrincipal Database Engineer HBS Information TechnologyJune 2011
![Page 2: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/2.jpg)
2
Evolution of Enterprise Data Integration
The Need for a Data Integration Framework • Why Change? • Steps Taken by HBS
Enabling Change • Where to Start?• Key Considerations
The Result • A Look Inside• Success Cases • Lessons Learned
Questions
Agenda
![Page 3: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/3.jpg)
3
An evolution away from point solutions coding. While traditional data interfaces are ‘hardcoded’ solutions, specific to one use. A data integration framework is intended to define the components of a reusable information architecture, whereby publishing and consuming updates across the Enterprise are based on common Business Entity Objects (Master data).
What’s changing in Enterprise Data Integration at HBS?
![Page 4: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/4.jpg)
4
Why develop a Data Integration Framework?Information is an asset – as long as you can share it consistently
![Page 5: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/5.jpg)
5
Traditional data interfaces are ‘hardcoded’ solutions, specific to one use…
Even experienced developers have trouble managing the data flow at this level
![Page 6: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/6.jpg)
6
1. Application upgrades and new releases cause significant data integration re-development cycles – Database to Database
2. Business process improvements difficult to implement without changing code – Hardcode Business Rules
3. Only “Matt” knows how that works – Single points of failure!
What’s your favorite problem?
Bottom Line:• We all pay the price in Time ($), Credibility and Quality
What are the warning signs for change?
![Page 7: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/7.jpg)
7
1. Being a Change Agent
o Recognize you have a problem
o Communicate and build a group of internal advocates
2. Build a Clear Case/DI Strategy
o Market this as an Enterprise level strategy change
o Remediating current problems is only a short-term goal. Plan and invest for the
future, or you might never get that chance “next year”
3. Create a Governance model
o Identify the HBS Groups that will influence, either directly or indirectly, the
requirements upon which the HBS Enterprise Data Integration Strategy will be
developed and deployed.
The steps taken at HBS towards change
![Page 8: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/8.jpg)
8
Enabling ChangeTurning a Roadmap into Reality
![Page 9: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/9.jpg)
9
PowerCenter (Advanced Edition)
• An enterprise data integration platform for accessing, discovering, and integrating data
from virtually any business system, in any format, and delivering that data throughout
the enterprise.
Some major options procured:
• Power Exchange for Web Services to act as a web services provider and consumer with 3rd party
systems by using predefined transformations.
• Real Time Option to service requests on demand, as opposed to batch, for web services provider
or other event based protocols.
• High Availability Option minimizes service interruptions during hardware and/or software outages
and reduces costs associated with data downtime.
• Data Cleanse and Match Option features powerful, integrated cleansing and matching capabilities
to correct and remove duplicate customer data.
Where did we start?…Informatica is at the core of the DI Framework
![Page 10: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/10.jpg)
10
Thinking about data as a Business Object
• Shared Business Entity Objects within the HBS Enterprise database are defined by a common
object XSD, as an abstraction of the base tables that make up that entity. Thereby they relieve
the Source applications from having to understand the direct database table interrelationships.
Designing a reusable Transformation Layer
• Mappings are the details of the transaction data handling (i.e. Transformations) that occur from
Source to Target. Examples of typical transformations can include derived data capture, complex
calculations, data validation and cleansing.
Define an Interface Protocol
• The primary role of the interface, or abstraction layer is to transform Source data inputs into a
common format for transaction processing. The format we chose to represent these data inputs
in a common form is XML.
From building blocks to a DI Framework
![Page 11: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/11.jpg)
11
Let’s take a look insideInformatica and HEDEx
![Page 12: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/12.jpg)
12
HBS Enterprise Data Exchange (HEDEx)
![Page 13: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/13.jpg)
13
When to use a DI framework Guidelines for your team
![Page 14: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/14.jpg)
14
When to use a DI framework
![Page 15: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/15.jpg)
15
External Relations – Alumni Data Management
• An enhanced reporting solution for Donor Behavior which interfaces with our external
Alumni clubs (Alumni Magnet) and data from the Harvard University Advance system.
Type – Bulk Data Load Publisher
• “I am so grateful and excited that we finally have these reports to guide and evaluate our
fundraising strategies” – Charlotte Smith, Director, Development Marketing and
Production
Executive Education – Program Delivery
• Transfers participant and program data, and enables the transition from enrollment
(CRM) to program delivery tools that are Enterprise-based applications. Type – Change
Data Publisher
• Reduced maintenance hours - 40% drop from the same review period in the previous year
and provides early notification of errors and self-service correction.
Start with small success stories to build upon
![Page 16: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/16.jpg)
16
The real value in a DI Framework!
![Page 17: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/17.jpg)
17
Form a Data Integration Special Interest Group to address
• Define standards around Business Entity Objects for DI use across the enterprise.
• Create a framework for documentation, review, development and consumption of DI
components.
• Draft a DI Strategy and Roadmap: A DI framework is a product like any other!
Not everything works the way its advertised
• Informatica is an enterprise level tool suite (e.g. not Toad for Oracle). The greater the
effort put into building the Mappings and Workflows the first time, the better the
opportunity for re-use. Caution: allow for the unexpected
Training is not optional
• While skill and determination will get you into a new technical tool suite, there is no
substitute for formal training.
Lessons we’ve learned so far
![Page 18: Building a Reusable Data Integration Framework](https://reader035.vdocuments.us/reader035/viewer/2022062315/5681617c550346895dd10c17/html5/thumbnails/18.jpg)
18
• Share Point Integration…
• Oracle PeopleSoft (Campus Solutions) Integration…
• Restate our HEDEx Roadmap…
So what’s next…