leader in data warehouse appliances

Dai Clegg

Leader in Data Warehouse Appliances

14 czerwca 2011 r.Warszawa, Sheraton Warsaw Hotel

The IBM Netezza Data Warehouse Appliance:

faster, simpler, more accessible analytics

The IBM Netezza Appliance: Revolutionizing Analytics

What is Netezza?

The IBM Netezza Appliance: Revolutionizing Analytics

Purpose-built analytics engine Integrated database, server & storage Standard interfaces Low total cost of ownership

Speed: 10-100x faster than traditional systems Simplicity: Minimal administration Scalability: Peta-scale user data capacity Smart: High-performance advanced analytics

IBM Netezza Appliance Overview

Customers Appliance Simplicity Appliance Architecture Advanced In-database Analytics Summary

Appliance Simplicity

Managing The Netezza Appliance

No storage administration

No database tuning

No software installation

Less DBA drudgery, More applications

Data Integration

Data In

The Netezza Appliance – Loading

Ab Initio

Business Objects/SAP

Composite Software

Expressor Software

GoldenGate Software (Oracle)

Informatica

IBM Information Server

Sunopsis (Oracle)

WisdomForce

SQ

L

OD

BC

JD

BC

O

LE-D

B

The Netezza Appliance – Querying

Reporting & AnalysisActuateBusiness Objects/SAPCognos (IBM)Information BuildersKalidoKXENMicroStrategyOracle OBIEEQlikTechQuest SoftwareSASSPSS (IBM)Unica (IBM)

SQ

L

OD

BC

JD

BC

O

LE-D

B

Data Out

Simple to Deploy and Operate Operations

Simply load and go Installation to Business Value in ~2 days

BI Developers No configuration, indexes or tuning

out of the box performance

ETL Developers Faster load and transformation times

simpler ETL logic & in-database transformation

Business Analysts Lower latency

load & query simultaneously True ad hoc queries

Customer Success

Digital Media

Financial Services

Government

Health & Life Sciences

Retail / Consumer

Products

Telecom

Other

“…when something took 24 hours I could only do so much with it, but when something takes 10 seconds, I may be able to completely rethink the business process…”

- SVP Application Development, Nielsen

15,000 users running 800,000+ queries per day 50X faster than before

Source:http://www.youtube.com/watch?v=yOwnX14nLrE&feature=player_embedded

Speed

http://www.youtube.com/watch?v=yOwnX14nLrE&feature=player_embedded

DAYS

WEEKS

MONTHS“Allowing the business users access to the Netezza box was what sold it.”

Steve Taff,

Executive Dir. of IT Services

Simplicity

200X faster than Oracle system

ROI in less than 3 months

Up and running 6 months before having any training

“NYSE … has replaced an Oracle 10 relational database with a data warehousing appliance from Netezza, allowing it to conduct rapid searches of 650 terabytes of data.”

ComputerWeekly.com

Source: http://www.computerweekly.com/Articles/2008/04/14/230265/NYSE-improves-data-management-with-datawarehousing.htm

Scalability

1 PB on Netezza

7 years of historical data

100-200% annual data growth

http://www.computerweekly.com/Articles/2008/04/14/230265/NYSE-improves-data-management-with-datawarehousing.htm

Smart

“Because of (Netezza’s) in-database technology, we believe we'll be able to do 600 predictive models per year (10X as many as before) with the same staff."

Eric Williams,

CIO and executive VP

Coupon redemption rates as high as 25%

Predicts what shoppers are likely to buy in future visits

Appliance Architecture

Server

CACHE

IBM Netezza True Appliance Architecture

SQL

DATA

Source Systems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

TRU64

SQL Data

Storage

CACHE

Database

CACHEI/O I/O


Source Systems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

TRU64

ODBC 3.XJDBC Type 4

SQL-92SQL-99 Analytics

Database, Server, Storage - in one

Storage

CACHE

Server

CACHE

Database

CACHE I/O I/O


Optimized Hardware+Software

Purpose-built for high performance analytics; requires no tuning

Streaming Data

Hardware-based query acceleration for blistering fast results

True MPP

All processors fully utilized for maximum speed and efficiency

Deep Analytics

Complex analytics executed in-database for deeper insights

Appliance family for data life-cycle management

COMING SOON

From a few terabytes to 10s of petabytes

Massively Parallel Processing

IBM Netezza True Appliance Massively Parallel Processing™

Massively Parallel Intelligent Storage

1

2

3

960

ŸŸŸ

Network FabricSMP Host

DBOSFront End

High-Speed Loader/Unloader

ODBC 3.XJDBC Type 4

OLE-DBSQL/92

Execution Engine

SQL Compiler

Query Plan

Optimize

Admin

Source Systems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

TRU64

High-PerformanceDatabase EngineStreaming joins,

aggregations, sorts

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic


aggregations, sorts

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic


1

2

3

960

ŸŸŸ


DBOSFront End


SQL Compiler

Query Plan

Optimize

Admin

SQL

Snippets

1 2 3

SQL

Source Systems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

TRU64


Execution Engine

Our Secret Sauce

FPGA Core CPU Core

Uncompress Project Restrict,Visibility

Complex ∑Joins, Aggs, etc.

select DISTRICT,

PRODUCTGRP,

sum(NRX)

from MTHLY_RX_TERR_DATA

where MONTH = '20091201'

and MARKET = 509123

and SPECIALTY = 'GASTRO'

Slice of table

MTHLY_RX_TERR_DATA

(compressed)

Slice of table

MTHLY_RX_TERR_DATA

(compressed)where MONTH = '20091201'

and MARKET = 509123


where MONTH = '20091201'

and MARKET = 509123


sum(NRX)sum(NRX)select DISTRICT,

PRODUCTGRP,

sum(NRX)

select DISTRICT,

PRODUCTGRP,

sum(NRX)


aggregations, sorts

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic



1

2

3

960

ŸŸŸ


DBOSFront End


SQL Compiler

Query Plan

Optimize

Admin

Consolidate

Source Systems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

TRU64

Execution Engine

Advanced Analytics

Advanced Analytics the Traditional WayAdvanced Analytics the Netezza Way

Fraud Detection

Demand Forecasting

SAS, SPSS

R, S+

AnalyticsGrid

DataWarehouse

C/C++, Java, Python, Fortran, …

Data

SQL

SQL

ETL

SQL

ETL

ETL

Phase I speed up the investigation & extract with the

Netezza warehouse

1) Extract data into analytic workbench or grid2) Develop model3) Test Model4) Score model against whole database5) Debug by discarding and iterating from step 1

Advanced Analytics the Netezza Way

ETL

SAS, SPSS

R, S+

Fraud Detection

Demand Forecasting

AnalyticsGrid

C/C++, Java, Python, Fortran, …

Data

SQL

Advanced Analytics the Netezza Way

SAS, SPSS

R, S+

SQL

SQL

Fraud Detection

Demand Forecasting

complex analytics SAS, SPSS, R, Java, etc

implicit parallelism petabyte scalability appliance simplicity

Phase II Move model functionality into the Netezza

warehouse Using its bult-in analytic libraries

Save costs and improve analyst efficiency

Summary

The IBM Netezza Appliance Purpose-built analytics engine Integrated database, server & storage Standard interfaces Low total cost of ownership

Digital Media

Financial Services

Government

Health & Life Sciences

Retail / Consumer

Products

Telecom

Other

Thank You

leader in data warehouse appliances

Documents

data warehousing appliance

netezza box

database analytics summary

terabytes of data

relational database

database technology

accessible analytics

oracle systemroi