information excellence iisc denodo data virtualization_20130420

Information Excellence informationexcellence.wordpress.com

Harvesting Information Excellence

Information Excellence

2013 Apr Knowledge Share Session


Community Focused

Volunteer Driven

Knowledge Share

Accelerated Learning

Collective Excellence

Distilled Knowledge

Shared, Non Conflicting Goals

Validation / Brainstorm platform

Mentor, Guide, Coach

Satisfied, Empowered Professional

Richer Industry and Academia

About Information Excellence Group

Progress Information Excellence

Towards an Enriched Profession, Business and Society


Host Us

Two Hour Monthly Session

Half Day deep dive Session

Full Day Summit Session

Speaker Support

Recommend Speakers

Suggest Topicss

Sponsor (In Kind, No Cash)

Welcome Kit

Speaker Mementoes

You Can Help this Community Grow

Progress Information Excellence

Towards an Enriched Profession, Business and Society

All Our Sessions are Free for participants; All Support and Sponsorship in Non Cash mode


2013 Apr Knowledge Share

Data Virtualization and Linked Data Services:

The foundation for Agile Business Intelligence,

Big Data and Dynamic Information Management

in the 21st century

Thank You for hosting us today


Suresh Chandrasekaran,

Senior VP, Denodo Technologies. Today’s

Speakers

Topic Abstract:

Data Virtualization has gone mainstream! It has been adopted as a

critical information management capability at major companies

worldwide and leading SI firms.

The need is driven by business users, BI analysts and application

developers wanting timely and agile data access to a broad array of

enterprise data assets, and new sources like Cloud, Big Data, NoSQL,

Web and Social Media.

Data Virtualization provides the ability to integrate disparate sources and

present them as Linked Data Services for both BI and operational use.

Can it Transform and Change the world of Data and Decisions?

Can it impact the Business Models?


Today’s Speakers

Topic Abstract:

Context and Need for Data Virtualization: How have data sources,

technologies and business needs changed?

What is Data Virtualization? What are the key capabilities and value it

provides?

Industry use cases for Data Virtualization

Analytical (Agile BI and Big Data Analytics),

Operational (Single View, Mobile & Cloud applications, Data Services, etc.)

Data Management (Logical Data Warehouse, Virtual MDM etc. ) scenarios

Data Virtualization / Linked Data Services – Live demo

Business Impact, ROI and Best practices for Data Virtualization

Q&A.

TOPIC ABSTRACT


Today’s Speakers

Suresh is a recognized expert in Data Virtualization.

Over 15 years of experience in leading integration middleware and

Web companies. Previously, a management consultant at Booz & Co

He speaks frequently at conferences on technology and its business

impact, most recently at Enterprise Data World, several regional

DAMA meetings, TDWI, Forrester EA forum, etc.

He has an MBA from the University of Michigan and undergraduate

degrees from India.

Used to be a CA and Financial Person, Mergers and Acquisitions

Enjoys Karnatik Music, his wife runs Bharatat Natyam dance school

Mid-career Travel (after Booz) – 2 yrs on motorcycle around the world

Suresh Chandrasekaran,

Senior VP, Denodo Technologies.


Today’s Speakers

Juan Lozano, Sr. Solution Consultant, Denodo

Over 7 years in the Data Integration and Data Virtualization software

industry with Denodo.

Deep experience in multiple business software implementation and

integration projects turning business needs into technical solutions.

Worked in several strategic multinational projects and interacted with

customers, at both CxO and technical levels.

Engineer (BS and MS) from Universidad Antonio de Nebrija, SPAIN

Continued Knowledge quest with UC Berkeley and Stanford

Supporting Presentation Roles

Suhaas Kodagali, Product Marketing, Denodo

B.Eng, BHU, MBA, Darden School, UVA; Worked at Cognizant in

Bangalore and USA; Works on Customer Case Studies & Marketing

http://www.linkedin.com/college/?eduSchool=12193&trk=prof-edu-school-name

http://www.linkedin.com/college/?eduSchool=12193&trk=prof-edu-school-name

© 2013 Denodo Technologies

Data Virtualization and Linked Data Services: The Foundation for Agile Business Intelligence, Big Data and Dynamic Information Management in the 21st century

Suresh Chandrasekaran [email protected]

April 20, 2013

mailto:[email protected]


Denodo thanks the Information Excellence Group and

IISc Bangalore for Hosting this Exclusive Session

We encourage comments and questions.

Please submit your Q’s to moderator with name of your org.

A recording of this presentation will be available online at

Information Excellence website and www.denodo.com

All content rights are reserved by Denodo Technologies, Inc.


Agenda

■ Data Challenges for Business and IT

■ Data Virtualization - Faster Access, Agile Provisioning

■ Industry Use Cases

• Analytical (Agile BI and Big Data Analytics, Logical DW, DW Prototype, etc)

• Operational (Single Customer View, Cloud Integration, Mobile Apps)

• Data Management (Data Services / IaaS, Unified Governance, Virtual MDM)

• Industry Specific Use Cases

■ BREAK: Data Virtualization Videos

■ LIVE DEMO: Data Virtualization / Linked Data Services

■ Summary - Business Impact & ROI

■ Q&A (At end of each session)


Key Industry Data Trends

■ Increasing data volumes

Tech innovations and new data collection methods like machine data, weather sensors, web logs, social media

■ Increasing data usage apps/devices

Multiple BI & analytic tools, cloud and mobile apps to track and use data

■ Increasing analytics processes

Increased analytics (self-service, big data, predictive) to gain competitive advantage

■ Business drivers for speed, agility, cost Faster time to market on new products and business models, compliance, M&A integration, cost efficiencies


Need consolidated view of the business entities to enable agile reporting, end users to discover /

browse data, and applications to reuse data services

Business Need

Customers

Agents Risk Factors

Products Claims


Real World

Multiple data silos (DB,

DW, Excel, Mainframes,

AS400, Big Data, NoSQL)

Multiple access needs (BI

tools, mobile, cloud apps)

Access to new data types

and external sources is hard

Data quality issues -

Redundant data, Direct back-

end updates, copies etc.

High legacy integration

and custom coding to

replicate data which is costly

and inflexible to changes

Agent Data (MS SQL Server)

Customer Data

(MySQL)

Claims System (Web Service -

Rest)

Auto location / smart grid data (Hadoop, NoSQL)

Ext. Information - D&B, Lexis Nexis

(Web data)

Product Data (CSV)

ERP (Sales, Contract) Data

(Cloud App.)

Pricing/Analytics (Web Application)

Legacy Data (Mainframe,

AS400)


Challenges

■ Data spread across heterogeneous data sources with different data formats

■ Canonical views must combine data from different sources

■ Data may need to be transformed and cleaned

■ Create the associations between the related entities

■ Ensure good performance

■ Need for rapid time to market, flexible to update / change

■ Minimize replication of data which is costly

■ Increase reuse of data (self-services, data services etc)

■ Integrated governance, security, data quality / master data


Data Virtualization

BASE VIEW

BASE VIEW

BASE VIEW

DERIVED VIEW

Data Virtualization combines disparate data sources into a single “virtual” data layer that provides unified access and integrated data services to consuming applications in real-time (right-time)

Connect &

Virtualize

Combine &

Integrate

Publish as a Data Service


Data Virtualization Delivers New Capabilities

Data Virtualization

Logical abstraction &

decoupling

On-demand, real-time access to

disparate data

Semantic integration of

structured, Web,

unstructured

Agile Data services

Provisioning

(many formats, secure, reusable)

Unified data governance


Data Virtualization Platform Architecture

Da

ta C

on

su

me

rs

Enterprise Applications, ESB Reporting, BI, Portals Mobile, Web, Users D

ata

Vir

tuali

za

tio

n

Design Tools

Optimizer

Cache

Scheduler

Monitoring

Governance

Metadata

Security

Publish Real-time (Right-time) Data Services

Combine Transform, Improve Quality, Integrate

Connect Normalized Views of Disparate Data

Denodo Platform

Library of Wrappers Web Automation Any Data or Content Read and Write

Da

ta S

ou

rce

s

Databases & Warehouses

Enterprise Applications

Cloud / SaaS Applications

XML, Excel, Flat Files

Big Data, NoSQL

Web 2.0 Soc. Media

PDF, Docs, Index, Emails

More Structured Less Structured

Business Solutions Access Information- as-a-Service

Denodo Platform Right Information at the Right Time

Disparate Data Any Source Any Format

Multiple Protocols,

Formats

Linked Data Services

Query, Search, Browse

Request/Reply,

Event Driven

Secure

Delivery


Data Virtualization Platform Architecture

Da

ta C

on

su

me

rs

Enterprise Applications, ESB Reporting, BI, Portals Mobile, Web, Users D

ata

Vir

tuali

za

tio

n

Design Tools

Optimizer

Cache

Scheduler

Monitoring

Governance

Metadata

Security

Publish Real-time (Right-time) Data Services

Combine Transform, Improve Quality, Integrate

Connect Normalized Views of Disparate Data

Denodo Platform

Library of Wrappers Web Automation Any Data or Content Read and Write

Da

ta S

ou

rce

s





Big Data, NoSQL

Web 2.0 Soc. Media



Business Solutions Access Information- as-a-Service

Denodo Platform Right Information at the Right Time

Disparate Data Any Source Any Format

Multiple Protocols,

Formats


Query, Search, Browse

Request/Reply,

Event Driven

Secure

Delivery

Universal Data Access

Agile High Performance

Universal Data

Publishing

Unified Data Governance

Unified Virtual Data

Layer


Financial: Return on Data Assets (ROA) 3

IT / Structural: Single Unified Data Layer 2

Business: Speed, Flexibility, Cost 1

Why Data Virtualization?

Data Virtualization enables timely delivery of data and flexibility to change at lower cost than traditional integration

A unified virtual layer that abstracts underlying source complexity and presents disparate data as if from a single source

Multiply business value by increasing ‘shareability’ of a data resource across multiple data users and use cases


Data Virtualization Use Cases


• Real-time Reporting

• Unified Dashboards

• Virtual Logical DW / Marts

• Big Data / Predictive Analytics

• Self-Service & Mobile BI

• Customer Single View - Call Center, Portals

• Data Services for BPM

• Operational Decision Data

• Unified Desktops - Underwriting, Loans,

• Big Data for Enterprise

• Cloud / SaaS Integration

• Web Data Aggregation

• Social Media, Feedback

• Multi-structured

• Linked Data Services (IaaS)

• Unified Data Layer

• Logical Data Abstraction – Migration / Modernization

• Data Mashups

• Unified Source of Reference - Virtual MDM

Agile Business Intelligence

Agile Single View Applications

Big Data, Cloud Integration


Denodo Data Virtualization – Use Cases

• DS for Mobile Apps


Biogen Idec - Agile BI Real-time Sales Reporting Across 90 Countries


Biogen Idec Key Benefits and ROI fromData Virtualization

Biogen Idec executives now have access to new partner and public data sources that they could not leverage before : 60% faster with change requests met in just a few days with IT using 40% less analyst time to support.

Business Problem • Manual process • Data freshness and quality

concerns • Fragile and complex to change Business Benefits • Automated • Validated and trusted data • Self service enabled for changes • Available on-demand

IT Challenges • Diverse data sources • Structured and semi-structured data • Data structure complexity • Constantly changing sources and

data IT Benefits • Implementation - 4 weeks • Changes – 2,4 days • Low maintenance and support costs

Biogen Idec Example Executive report – Global Drug Sales (units) across all products


Operational BI – Unified Dashboards Data Services for Dashboards, Performance Management


Operational BI – Unified Dashboards Data Services for Dashboards, Performance Management

Incidences Summary

Classification Ranking

Configuration (CMDB) Changes


Presentation at Buckeye DAMA (pre-production phase): http://buckeyedama.org/events/past-presentations/

http://buckeyedama.org/events/past-presentations/




Conformance to Industry Standard Models

ACORD Industry Models

XML Models imported as

Interface views

Interface Views (Virtual Models

without implementations)

Virtual business entities

Normalized base views of data

sources

Person Claims Policy Premium

Source data spread across multiple source systems

Person Claims Policy Premium

Data Virtualization

Data Virtualization allows easy adoption of industry standard models

• Top down

modeling to enforce contracts

• Creation of

logical data entities (data services)

• Agile development across multiple teams


Jazztel - Agile Customer Interaction Read / Write Data Services for Unified Agent Desktops, Customer Portals

Deliver Customer Information-as-a-Service to multiple front-ends Transform and combine into Virtual Data Views based on context and agent profile Access all data: 51+ sources back-end systems, partner, content and external web Support write-backs through DV / Data Services Platform or directly to Transaction systems 2500 contact center agents, High concurrency > 1 million transactions a day 25% cache hit rate, High-performance active –active cluster configuration

Use of Data Virtualization

Using Data Virtualization to enable Agile BI

Bhavna Kapoor

BI Architect, The Climate Corporation

February 2013

Case Study Presented at TDWI Conference in Las Vegas, NV. Download from: http://www.denodo.com/en/resources/documentation/case_studies/case_study_the_climate_corporation.pdf

http://www.denodo.com/en/resources/documentation/case_studies/case_study_the_climate_corporation.pdf

ANALYSIS

- Aggregating/analyzing very large dynamic data sets (Platform)

- Modeling weather’s impact on crop yield (Agronomy)

- Simulating weather to assess risk, recommend coverage and price

policies (Climatology)

OUR CORE COMPETENCIES

SIMULATION

- 30 T simulation

values

- 34 TB of

simulations

saved in S3

- Unique

statistical

models

AGGREGATION - 1000s of

concurrent

instances

running map-

reduce clusters

- Unique agronomic

research

(machine

learning

- Daily weather

measurements for

over 500K

locations

- 712 unique

measurements

- 22 weather

datasets

ingested every 6

THE BI CHALLENGE

Data needed to answer key business questions is scattered across disparate systems in disparate forms

Data Integration

Traditional ETL / Data Warehouse Abstracted logical views of

enterprise data

Strengths

• High scale, batch oriented data

delivery

• Complex transformations

• Query performance

Challenges

• Multiple copies of the data

• Time-to-market

Strengths

• Unified, on-the-fly access to

heterogeneous data sources

• Push-down query optimization

• Intelligent caching

• Faster, agile development

Challenges

• High source resource utilization

• Limited transformation

FACTORS & CONSIDERATIONS

• Source data sets are constantly changing – different models, storage technology, formats

• Reporting requirements are fairly straightforward (aggregations, slicing-and-dicing) but requirements change frequently

• Avoid large scale data replication – huge overhead to build, keep in sync, storage costs

• Move towards RESTful web interfaces – data as a service

BI Platform Architecture

Impact

Data Virtualization

• First use-cases for Sales implemented

• 3X improvement in time-to-market with 1/3rd the team

• Other use-cases

• Risk reporting automation using Hive/Denodo

• Linked data services and BI Portal

• Operational Data Integration


• Recognized that its highest risk comes from dynamic areas of medicine with rapid advances in technology

• Used Data Virtualization to integrate unstructured big data sources with 25 years of claims history data and internal sources

• Leading Patient safety and medical malpractice insurance company serving a large university medical community since 1976 - - Over 10000 doctors, 18 hospitals and 238 other healthcare organizations.

Unstructured Intelligence for Clinical Quality


Virtualized Data Services Reduce Clinical Risk

EDW - Claims, Payments

Risk Management Applications

For Medical Malpractice Insurance

Risk Assessment &

Pricing

Physician Best Practices

Public Web - MHQP, MA Med board …

Claims Knowledge Base

Risk Models, Polices


Location Based Services Past, Present, Future


Navteq - Agile Location Based Services

Combine EDW, Cloud, data feeds and real-time social media

Batc

h Data Virtualization

Real-time Integration Web & Social Media

Batch

Real Time

Location Content Management System

Smart API – Core Widgets

User Data Store Service

Web, Devices | Users | Partners | App Builders

POIs


Agile Location Based Services - Combined View of Static and Dynamic Data


DV Vision = Unified Virtual Data Layer Enterprise Abstraction, Semantic Data Services, Agile Everything

Replicate

MDM/ Data warehouse

ETL

Propagate

SOA/ Service Registry

BPM/ESB

Collaborate

Portals, File Sharing

Emails, Social Platforms

Search

CMS Knowledge-db

Crawl, Index, Semantic Tool

Federate

Real-time Query Engine

Connectors

Agile BI &

Analytics

Agile Applications

People, Processes

Content Intelligence





Big Data NoSQL

Collaboration Web 2.0



Diverse Data Sources and Formats

Data Virtualization

Data Virtualization


Solve one/more projects

Prove value

Expand to other use

cases

Reusability and

scalability

Expand enterprise-

wide

Enterprise strategic

vision

Design for scale and support

Systemic data

services

High value use cases

first

Multiple usage

patterns

Tactical Approach

Enterprise-wide Approach

People Process Technology

Hybrid Approach Possible

Enterprise-wide Approach to Data Virtualization


Oppenheimer Funds – Enterprise Strategy For Turning Data Into Informaiton

Data Virtualization

Enterprise Dynamic Information Systems

RESTful Linked Data Services – all data assets represented as a resource …

easy to search, browse, query …


Data Virtualization at Oppenheimer

■ Access any source

■ Make them look homogeneous

■ Pre-integrated graph of information for users

■ Build very rapidly


Data Federation

DW Extension

Enterprise Data Sharing

Real-time Enterprise

Cloud Integration

Virtual MDM

MDM Registry

Data Virtualization Patterns


Data Virtualization Deployment Benefits & ROI

■ Easier to transform and combine disparate data sources • Unified, real-time access for consuming applications

■ Linkable, browsable data through RESTful access

■ Data is not copied, so reduces redundancy and is ‘timely’

■ Rapid provisioning of virtual data sources

■ Variety of delivery interfaces • JDBC/ODBC, REST/SOAP services, XML/JSON/HTML

■ Caching to improve performance

• Request-level, Fragment-level, View-level, etc.

■ Lots of DV patterns in use makes it a great utility asset

ROI: Denodo Data Services architecture enabled us to save thousands of programming hours in recent portal and application deployments.

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79 Cisco Confidential Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 79

Cisco Services Data Virtualization For Smart Services

Sanjeev Ukhalkar Senior Director, Cisco Services November 2012

80 80

Business

Analytics

Application/

Social

Analytics

Network/

Infrastructure

Analytics

Analytics focused on Business

Services Management, Cost

Optimization, Business Optimization,

Service Quality

Application/Server/Virtualization

related analytics that help optimize

various metrics like Performance,

Capacity Planning, Availability

Analytics around network, optimizes

networks for Performance, Capacity,

Fault, Availability, Risk, Compliance

Cisco Smart

Services Platform

• Node-based

architecture with

data aggregation,

virtualization sets

stage for predictive

analytics

• Analytics algorithms

leverage data from

multiple sources—

Big Data

• Virtualization,

dynamic

provisioning,

horizontal scaling

support delivering

business insight

85 85

Cisco Smart Services Platform

Enterprise Apps Reporting, Portals Mobile / Web

Professional

Services

Support

Services Operate

Services

Customer Support

(web services)

Cisco Network Inventory,

Contracts, Alerts

Service

Requests

Authorization Services

(web services)

Da

ta S

ou

rce

s

Cu

sto

me

rs,

Pa

rtn

ers

86 86



Professional

Services

Support

Services Operate

Services

Customer Support

(web services)


Contracts, Alerts

Service

Requests


(web services)

Da

ta S

ou

rce

s

Cu

sto

me

rs,

Pa

rtn

ers

• Difficult to flexibly combine disparate data sources – unforeseen needs to add new, emerging, less-structured data sources to traditional ones

• Need an abstraction layer-- data services / data asset strategy that lets analytics and UI layers focus on what they do best

• How to handle hybrid of persistent data + real-time, master data + transaction data

• Need dynamic partitioning and clusters to increase resource utilization, horizontal scalability

• Higher development costs

87 87



Professional

Services

Support

Services Operate

Services

Customer Support

(web services)


Contracts, Alerts

Service

Requests


(web services)

Da

ta S

ou

rce

s

Cu

sto

me

rs,

Pa

rtn

ers

Data Aggregation / Data Virtualization Layer

89 89



Professional

Services

Support

Services Operate

Services

Customer Support

(web services)


Contracts, Alerts

Service

Requests


(web services)

Da

ta S

ou

rce

s

Cu

sto

me

rs,

Pa

rtn

ers

Data Aggregation / Data Virtualization Layer

• Provides data integration, aggregation from all sources

• Delivers virtual data services

• Engineering costs reduced up to 25%

90 90

Data Virtualization drives: Integration, aggregation of data from multiple

sources—for virtual data services

Dynamic, horizontal scaling—to meet current and emerging market demands

Better performance, lower cost


• Real-time Reporting

• Unified Dashboards

• Virtual Logical DW / Marts

• Big Data / Predictive Analytics

• Self-Service & Mobile BI

• Customer Single View - Call Center, Portals

• Data Services for BPM

• Operational Decision Data

• Unified Desktops - Underwriting, Loans,

• Big Data for Enterprise

• Cloud / SaaS Integration

• Web Data Aggregation

• Social Media, Feedback

• Multi-structured

• Linked Data Services (IaaS)

• Unified Data Layer

• Logical Data Abstraction – Migration / Modernization

• Data Mashups

• Unified Source of Reference - Virtual MDM

Agile Business Intelligence

Agile Single View Applications

Big Data, Cloud Integration


Denodo Data Virtualization – Use Cases

• DS for Mobile Apps


Agenda

■ Data Challenges for Business and IT

■ Data Virtualization - Faster Access, Agile Provisioning

■ Industry Use Cases

• Analytical (Agile BI and Big Data Analytics, Logical DW, DW Prototype, etc)

• Operational (Single Customer View, Cloud Integration, Mobile Apps)

• Data Management (Data Services / IaaS, Unified Governance, Virtual MDM)

• Industry Specific Use Cases

■ BREAK: Data Virtualization Videos

■ LIVE DEMO: Data Virtualization / Linked Data Services

■ Summary - Business Impact & ROI

■ Q&A (At end of each session)


Key Takeaways

■ Data Virtualization is mainstream

• Many companies implementing Unified Virtual Data Layer to enable abstracted, decoupled, reusable data services

■ Costly and uneccesary replication can be avoided

• Data Virtualization allows data to remain in place and can reduce overall maintenance costs.

■ Development agility enhanced across data silos

• Building and publishing data services allows agile development and prototyping

■ Managing data access and quality is easier

• Data Virtualization is “unified source of reference” for discovering, managing, and auditing data access with fine-grained security / authorization to access data assets


Results You Can Expect from DV

Better Quality Information Focus on Business Information Needs Integrate Disparate Data Silos Include Web / Cloud, Big Data, Unstructured

Lower Cost & Agility Lower Integration Costs by 80% Flexibility to Change Real-time (on-demand) Data Services

Fast Time to Solution Projects in 4-6 Weeks ROI in <6 months Adds New IT and Business Capabilities


Denodo – Focused Leader in Data Virtualization

Forrester - Leader in Forrester WaveTM: Data Virtualization

Gartner - Cool Vendor in Data Integration

Bloor – Top 10 IT Companies to Watch

The Burton Group – Best of Breed Data Virtualization

InfoWorld - Best Mashup Server

Information Management - Top 40 List for 2011

BioIT World- Best of Show 2011-12

Broadest Support for Data Sources, Highest Performance, Most Flexible Delivery, Agile Linked Data Services, Increased ROA


Data Virtualization – More Resources

• White Papers: High Value Uses of Data Virtualization- Analytical, Operational, Data Management

http://go.denodo.com/datavirtualization_use_cases

• World Series on Data Virtualization - Numerous Case Studies in 3 Languages

http://www.denodo.com/en/resources/webcasts /dv_world_series_2011/index.php

• Expert’s Perspectives - Analyst and Expert Webinars

• Evaluation of Denodo Platform - Watch Online Demo - Pilot Evaluation Version - Partner Program






information excellence iisc denodo data virtualization_20130420

Technology

data integration

data sources

world of data

scenarios data virtualization

data virtualization

linked data services

agile data access

big data analytics