ukoug tech 15 - odi lifecycle and data governance

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Jerome Francoisse | UKOUG Tech 2015

Oracle Data Integrator – Development Lifecycle & Data Governance

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Jérôme Françoisse• Consultant for Rittman Mead- Oracle BI/DW Architect/Analyst/Developer• ODI Trainer

• Providing ODI support on OTN Forums• ODI 12c Beta Program Member• Blogger at http://www.rittmanmead.com/blog/• Email : jerome.francoisse@rittmanmead.com• Twitter : @JeromeFr

info@rittmanmead.com www.rittmanmead.com @rittmanmead

About Rittman Mead

•World’s leading specialist partner for technical excellence, solutions delivery and innovation in Oracle Data Integration, Business Intelligence, Analytics and Big Data

•Providing our customers targeted expertise; we are a company that doesn’t try to do everything… only what we excel at

•70+ consultants worldwide including 1 Oracle ACE Director and 2 Oracle ACEs

•Founded on the values of collaboration, learning, integrity and getting things done

Optimizing your investment in Oracle Data Integration

•Comprehensive service portfolio designed to support the full lifecycle of any analytics solution

•4 UKOUG Partner of the Year Award 2015

info@rittmanmead.com www.rittmanmead.com @rittmanmead

User Engagement

Visual Redesign Business User Training

Ongoing SupportEngagement Toolkit

Average user adoption for BI platforms is below 25%

Rittman Mead’s User Engagement Service can help

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Successful ODI project

• Having good developers is great• … but it’s not enough!

• We also need :- Assurance that the code deployed in production is

correct- The right data- Good data

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Agenda

• Lifecycle Management- Architecture- Promotion- Versioning • Data Governance- Oracle Enterprise Metadata Management- Oracle Enterprise Data Quality

info@rittmanmead.com www.rittmanmead.com @rittmanmead

ODI 12c Key Components

Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository(Development)

Execution

Execution Repository(Production)

• ODI Studio (the main development IDE)- Designer Navigator- Operator Navigator- Topology Navigator- Security Navigator• ODI Agents (executing mappings and

packages)• ODI Console (web-based interface)• ODI Repositories (contain definitions of

models, mappings and other objects)

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Physical and Logical Topology

DEV_SCOTT

SH

HR

SCOTT

HR

SCOTT

SH

HUM_RES

DEV_OP

TEST_OP

PROD_OP

Physical

SCOTT

HR

Contexts

Logical

DEV

TEST

PROD

SH

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Architecture

Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository(Development)

Execution

Execution Repository(Production)

info@rittmanmead.com www.rittmanmead.com @rittmanmead

How many repositories do I need?

Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository(Development)

Execution

Execution Repository(Production)

Execution

Execution Repository(Test)

info@rittmanmead.com www.rittmanmead.com @rittmanmead

How many repositories do I need?

Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository(Development)

Execution

Execution Repository(Production)

Execution

Execution Repository(Test)

Security

Topology

Versioning

Master Repository

info@rittmanmead.com www.rittmanmead.com @rittmanmead

How many repositories do I need?

Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository(Development)

Execution

Execution Repository(Production)

Execution

Execution Repository(Test)

Security

Topology

Versioning

Master Repository

Execution

Execution Repository(Pre-Production)

Security

Topology

Versioning

Master Repository

info@rittmanmead.com www.rittmanmead.com @rittmanmead

How many repositories do I need?

Models

Projects

Execution

Work Repository(Development)

Execution

Execution Repository(Production)

Execution

Execution Repository(Test)

Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository(Hotfix)

Execution

Execution Repository(Pre-Production)

Security

Topology

Versioning

Master Repository

Security

Topology

Versioning

Master Repository

info@rittmanmead.com www.rittmanmead.com @rittmanmead

How many repositories do I need?

Models

Projects

Execution

Work Repository(Development)

Execution

Execution Repository(Production)

Execution

Execution Repository(Test)

Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository(Hotfix)

Execution

Execution Repository(Pre-Production)

Security

Topology

Versioning

Master Repository

Security

Topology

Versioning

Master Repository

Execution

Execution Repository(Smoke Test)

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Problems• As a developer, how do I create a consistent set of objects for a

release?- For development environments- For execution environments

• As a release manager, how do I easily promote code from one environment to another?• As a release manager, how do I easily keep track of these sets of

objects?• As a release manager, how do I roll back to a previous release?

info@rittmanmead.com www.rittmanmead.com @rittmanmead

The Solution – Deployment Archive (DA)

• Archive file containing a set of consistent ODI objects• Applied to Deployed Repositories (test/production)• 3 types :- Initial- Patch- Rollback• Development or Execution

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Example

Development environment

QA environment (Deployed repository)

Developer

Release Manager

Bug

Creates Initial Release 1.0.0.0

Initialize repository with Release 1.0.0.0

Creates Patch Release 1.1.0.0

to add a features

Apply Patch Release 1.1.0.0

and create rollback DA

Apply Rollback Release 1.1.0.0

Creates Patch Release 1.1.0.1

to fix the previous bug

Apply Patch Release 1.1.0.1

and create rollback DA

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Developer / Administrator

• Create a deployment archive

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Developer / Administrator

• Create a deployment archivefrom VCS label

• Best Practice :Link documentation and DDL statements to be applied

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Developer / Administrator

• Create a deployment archive from selected objects

• Best Practice :Mention the bug number in the description

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Developer / Administrator

• No need to think about development or execution-only environment

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Release Manager

• Apply Deployment Archives

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Release Manager

• Initial DA• Can choose from

development or execution-only archive

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Release Manager

• Patch DA• Can choose from

development or execution-only archive

• Best Practice :Always create a Rollback DA!

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Release Manager

• See applied DAs – Inventory

• See available DAs – View Details

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Release Manager

• Rollback DA

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Automation

• Available through ODI SDK

• Scenarios :- Import and be sure you always generate a rollback

DA- Import daily into a smoke test environment- Import the same release in Hotfix and Production at

the same time

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Other Problems

• How to rollback on the Development environment?• How to restore delete objects in the Development

environment?

info@rittmanmead.com www.rittmanmead.com @rittmanmead

The Solution - VCS

• Integration with Subversion (SVN)- Everything is done through ODI Studio- Git support in the next release- Tagging- Branching / Merging

info@rittmanmead.com www.rittmanmead.com @rittmanmead

VCS - Features

• Add an object to VCS

info@rittmanmead.com www.rittmanmead.com @rittmanmead

VCS - Features• Create a new version of an object

info@rittmanmead.com www.rittmanmead.com @rittmanmead

VCS - Features• View and compare versions

info@rittmanmead.com www.rittmanmead.com @rittmanmead

VCS - Features• Restore a version

info@rittmanmead.com www.rittmanmead.com @rittmanmead

VCS - Features• Tags- Snapshot of consistent set of versioned objects- Ready for promotion – Deployment Archive- … or for branching- Created from

‣ Whole repository (Full Tag)‣ Selected Objects and dependencies (Partial Tag)

info@rittmanmead.com www.rittmanmead.com @rittmanmead

VCS - Features• Branches- Created from a Tag- Populate a new environment – Master and Work

repository- For a new feature or a new team- Can be merged back in the trunk

info@rittmanmead.com www.rittmanmead.com @rittmanmead

VCS - Features• Branches

Trunk

Branch 1

BranchingRelease 1

101 2 3 5

4 6 7 9

Merging

11

Bug Fixes for Release 1

New features for Release 2

New features for Release 2New features for Release 1

8

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Data Governance• The right data• The good data

People

Process Tools

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Data Governance• The right data• The good data

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Data Governance• The right data

Which reports usesmy application

data?

Application User Sysadmin

Which reports use the mainframe

data?

ETL Developer

What is the impact if I change this

mapping?

DBA

Who is using this database I need to

upgrade?

Data Steward

Where is this data entered?

Who should I notify if there is a

problem?

Which data are available?

Data Scientist BI Developer

Where does this data come from?

Executive

How are sales figures

calculated?

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Oracle Enterprise Metadata Management• Lineage

- From reports to source• Impact Analysis

- From source to reports• DB - ETL - BI Metadata• Business Glossary• Semantic Linking

• Supporting features- Harvesting- Automatic Stitching- Versioning- Collaboration - Annotations

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Oracle Enterprise Metadata Management

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Oracle Enterprise Metadata Management

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Oracle Enterprise Metadata Management

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Oracle Enterprise Metadata Management

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Oracle Enterprise Metadata Management

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Oracle Enterprise Metadata Management

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Data Governance• The right data• The good data

info@rittmanmead.com www.rittmanmead.com @rittmanmead

• The good dataOracle Enterprise Data Quality

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Oracle Enterprise Data Quality - Profiling

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Oracle Enterprise Data Quality - Cleansing/Matching

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Oracle Enterprise Data Quality - Cleansing/Matching

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Oracle Enterprise Data Quality - ODI Integration

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Summary• ODI - Deployment Archives- Promote, Rollback and keep track of your code- For development or execution-only environment- Easy for Developers and Release Managers• ODI - VCS- Be able to get back to any version to object- Branching for Team development• OEMM- Lineage, impact analysis and business glossary• OEDQ- Profile, Cleanse, Match, Monitor, Govern

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Further Reading / Testing• ODI Lifecycle- Managing versions in SVN

https://blogs.oracle.com/dataintegration/entry/oracle_data_integrator_12_21- Parallel Development using SVN Branches

https://blogs.oracle.com/dataintegration/entry/oracle_data_integrator_12_22- ODI Getting Started VM 12.2.1

http://www.oracle.com/technetwork/middleware/data-integrator/odi-demo-2032565.html

• OEMM- Oracle By Example

http://www.oracle.com/technetwork/middleware/oemm/learnmore/index.html

• OEDQ Getting Started 12.2.1http://www.oracle.com/technetwork/middleware/oedq/downloads/edq-vm-download-2424092.html

info@rittmanmead.com www.rittmanmead.com @rittmanmead

Questions?