denodo datafest 2016: what’s new in denodo platform – demo and roadmap

O C T O B E R 1 8 , 2 0 1 6 S A N F R A N C I S C O B A Y A R E A , C A

#DenodoDataFest

RAPID, AGILE DATA STRATEGIESFor Accelerating Analytics, Cloud, and Big Data Initiatives.

What’s New in Denodo Platform

Dr. Alberto Pan

Denodo, CTO

Agenda1. Performance

2.Self-Service

3.Managing Large Deployments

4.Connectivity

5.Q & A

What’s New

Some Recent and Upcoming Features

Main Areas

Dynamic Query Optimizer for Big Data (Denodo 6)

Incremental queries (Denodo 6 Updates)

Embedded in-memory fabric (Denodo 7)

New Information Self-Service Tool (Denodo 6)

Information Self-service: Glossary and Collaboration Features (Denodo 7)

▪ Tighter integration with Data Governance and

Data Modeling Tools (Denodo 7)

Workload Management: Denodo Resource Manager (Denodo 6)

Monitoring and Diagnostic Tool (Denodo 6 Updates)

Solution Manager (Denodo 7)

New VDP Admin Tool (Denodo 6)

GIT Support (Denodo 6)

▪ Support for new data sources and publishing formats (continuous work)

▪ New Data Types (Denodo 7)

Performance in BigData Scenarios

Security, Governance and Self-service

Enterprise Wide Deployments

Connectivity and Data Transformation

Move Processing to the Data

Process the data where it resides

Process the data locally where it resides

DV System combines partial results

Minimizes network traffic

Leverages specialized data sources

7

How to Choose the Best Execution Plan?

Cost-Based Optimization in Data Virtualization

Data statistics to estimate size of intermediate result sets

Data Source Indexes (and other physical structures)

Execution Model of data sources: e.g. Parallel Databases VS Hadoop clusters VS Relational Databases

Features of data sources (e.g. number of processing cores in parallel database or Hadoop Cluster)

Data Transfer rate

Must take into account:

8

Denodo has done extensive testing using queries from the standard benchmarking test

TPC-DS* and the following scenario

Compares the performance of a federated approach in Denodo with an MPP system where

all the data has been replicated via ETL

Customer Dim.2 M rows

Sales Facts290 M rows

Items Dim.400 K rows

* TPC-DS is the de-facto industry standard benchmark for measuring the performance of decision support solutions including, but not limited to, Big Data systems.

vs.Sales Facts290 M rows

Items Dim.400 K rows

Customer Dim.2 M rows

Denodo 6.0 ArchitecturePerformance Comparison – Logical Data Warehouse vs. Physical Data Warehouse

http://www.tpc.org/tpcds/

9

Denodo 6.0 Architecture

Query DescriptionReturned

RowsTime Netezza

Time Denodo (Federated Oracle,

Netezza & SQL Server)

Optimization Technique (automatically selected)

Total sales by customer 1,99 M 20.9 sec. 21.4 sec. Full aggregation push-down

Total sales by customer and year between 2000 and 2004

5,51 M 52.3 sec. 59.0 sec Full aggregation push-down

Total sales by item brand 31,35 K 4.7 sec. 5.0 sec. Partial aggregation push-down

Total sales by item where sale price less than current

list price17,05 K 3.5 sec. 5.2 sec On the fly data movement

Performance Comparison – Logical Data Warehouse vs. Physical Data Warehouse

10

Incremental QueriesNew Caching Mode for SaaS Data Sources

Merge cached data with delta changes from the data source

Real-time results with minimum latency

Data source needs to provide a way to obtain the delta changes

Get Leads Changed / Added since 1:00AM

CACHELeads updated at 1:00AM

Up-to-date Leads data

Full Cache – Incremental queriesConfiguration

1. Cached data

3. Merged based on PK

2. New data from source

11

12

Parallel In-Memory FabricEmbedded in-memory fabric fully integrated with cost optimization (Denodo 7)

Embedded in-memory fabric

MPP processing of costly local processing operations

External in-memory fabrics supported

Integrated with cost-based optimization

Main Areas



















14

Information Discovery and Self-Service (1)

Graphically Expose Data Views to Business Users

Search and Query Data andMetadata

Browse data associations

Transform and combine views

Publish results to Denodo or your favourite reporting tool

Find more details at: datavirtualization.bloghttp://www.datavirtualizationblog.com/data-exploration-and-self-service-bi-welcome-to-the-dataweb/

15


Browse associations between data views

16


Inspect Data Lineage

17


Search Content in All Views

18

Information Discovery and Self-Service (and 5)

Query, Combine and Transform Data Views

19

Information Self-Service Tool: 6.0 UpdatesEnhancements in 6.0 Updates

Support for Solr, Elastic Search in Global Search

See folders structure

See web services

Improved metadata search And Support for specifying field descriptions

20

Information Self-Service Tool: Denodo 7 (1)Extended metadata and Components Catalog

Categorized/Tagged catalog of data components to associate views and business terms

Extended metadata fields

Ability to Edit Metadata

21

Information Self-Service Tool: Denodo 7 (and 2)

Governance and Collaboration Features

Publish / share new components to the catalog

Governance:

- Approval process

- Stewards

Public and private comments

Main Areas



















23

Denodo Resource Manager

Controlled Resource Allocation

1 Defines a rule that will be triggered for “app1” and users with the role “reporting”

2 For those request that fulfill the rule, if the CPU usage is greater than 85%, will apply the following:• Reduce thread priority• Reduce the number of concurrent requests• Limit the number of queued queries

24

Monitor current state of servers and clusters

Inspect sessions, queries(with real-time trace), connections,...

Inspect data sources activity, cache load processes and content,...

Monitoring and Diagnostic Tool (1)Graphical Monitoring and Diagnosing of Servers and Clusters

Go back in time to the moment where a problem happened

Diagnose root cause of the problem

25

Monitoring and Diagnosing Tool (2)Graphical Monitoring and Diagnosing of Servers and Clusters

State: Summary of the state of the server/environment

Resources: physical resources (memory, cpu,…)

Requests: including real-time execution trace

Session: Currently opened sessions, including client application

Cache: cache load processes, cache contents,...

Datasources: pools state, active requests,...

Threads: priorities, CPU usage,...

Errors: Inspect logged errors and warnings

… and many others

Filter and sort information by any criteria

26

Monitoring and Diagnosing Tool (3)

Automatic Alerts (Denodo 6.0 Updates)

Server down

Data Source or Cache Down

% CPU Usage

Connection Pool full

…

Alerts (Visual / E-Mail):

27

Monitoring and Diagnostic Tool (and 4)Pre-defined Reports (Denodo 7)

Pre-defined graphical usage reports)

• Workload breakdown by application

• Most used views

• Requests per Data Source

• …

28

28

Denodo Solution ManagerMake it easier to manage large Deployments (Denodo 7)

Catalog of all elements of a Denodo deployment

Manage licenses configuration, logs and extensions

Automate migrations

Integrated governance workflows

29

Automate Migration Between Environments

Overview of the Migration Process in Denodo 7 (Simplified)

S11

denodo-prd-1

S21

denodo-prd-2

S12

S22

S13

S23SolutionManager

Properties DB

Developers Migration Admins

Development

Production

1. SelectElementsto Migrate

2. ValidateRevision

VCS

4. Deploy Revision

5. Save full VQL after Revision

Load Balancer

3. RegisterRevision

Main Areas









Monitoring and Diagnosing Tool (Denodo 6 Updates)










Multiple Tabs

MultipleDatabases

New VDP Admin Tool (1)

31

New VDP Admin Tool (and 2)

Collapsable Work Areas

32

33

New adapters for Spark, Redshift and Snowflake (already available), Presto DB (Q1 2017), Neo4j (Denodo 7)

New adapters for Denodo in IBM Cognos and Looker (already available), Tableau (Q4 2016)

Extended set of geospatial functions and GeoJSON support (Denodo7)

Continuous work on transformation functions

Connectivity:

Other Enhancements

Transformation / Integration:

Thank you!

© Copyright Denodo Technologies. All rights reservedUnless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.

O C T O B E R 1 8 , 2 0 1 6 S A N F R A N C I S C O B A Y A R E A , C A

#DenodoDataFest

denodo datafest 2016: what’s new in denodo platform – demo and roadmap

Data & Analytics