real-time data warehousing with sap netweaver business intelligence

7/22/2019 Real-Time Data Warehousing with SAP NetWeaver Business Intelligence

1/39

HTWG: AK-SWT Treffen 10.11.2006

Real-Time Data Warehousingwith

SAP NetWeaver BusinessIntelligence

Rainer Uhle, SAP NW BI Produktmanagement


2/39

SAP AG 2006, HTWG AK SWT 101106 / 2

Systemlandschaftsbetrachtung zum Thema Integration

BI

(OLAP)

ERP(OLTP)

Abbildung von Prozessen

FI/CO Einkauf, Fertigung, Vertrieb

HR

Analyse von Prozessen


HR

Ziel

Entscheidungsuntersttzung (DS)

Planung

ERM(RDBMS)

MDM

(M/R-DBMS)

Stamm- und Prozessdaten

Replikation

periodisch ETL Real-time Data Acquisition

stetig

EAI

sonstig

Direct Access

SAP

Terminologie

SAP

Terminologie
http://localhost/var/www/apps/conversion/tmp/scratch_7/50_BI_Consumer_Examples.sim


3/39


RAS Core Research Note G00130597, Bill Gassman, 30 September 2006

Options Proliferate for Real-Time Data Integration Technology

Numerous applications are emerging for real-time or low-latency

data integration. Consequently, IT development and operations

managers must identify the most cost-effective and appropriate

offerings to satisfy on-demand and event-driven needs.

Analysis:

Real-time data integration is the technology and process that providesapplication access to up-to-date information to meet time-sensitive business

requirements.

Technology Options

direct access

real-time data acquisition publish-subsribe model, change data capture, message exchange

master data centralization

File transfer protocol (FTP)


4/39


Challenges of Real-time Data Warehousing

Looking beyond traditional ETL -

expand the concept of data acquisition in order to reduce data

latency

Latency

time lag between an activity completion in an ERP environment

and the availability of the completed activity data in a state of

the art data warehouse environment

Latency = zerodemand for tools that enable direct access of data and

information without any latency across platforms

SAP NetWeaver 2004s BI capabilities for

Real-time Data Warehousing

Real-time Data Acquisition (RDA)

Direct Access


5/39


ETL, Flavours of ETL, Alternatives

ETL (Extraction, Transformation, Load)

New DataSource Concept with SAP NW2004s BI

DataSource Transformation Data Transfer Process (DTP)Flavours of ETL

Classical Staging Processes


Alternatives

Direct Access

How does the

Implementation of

ETL Processeslook like in an

SAP NetWeaver 2004s BI

Environment?


6/39

Real-time Data Warehousing - Introduction

Overview NW 2004s BI - ETL Capabilities

Real-time Data Warehousing with SAP NW 04s BI

Direct Access


Outlook and Summary


7/39




Direct Access


Outlook and Summary


8/39 SAP AG 2006, HTWG AK SWT 101106 / 8

ETL, Flavours of ETL, Alternatives



DataSource Transformation Data Transfer Process (DTP)Flavors of ETL


Persistency of transactional and master data in BI

Extraction using DataSources with services like packaging and sequencing,

delta handling

Transformations with graphical UI and sophisticated formula builder

Data loads based on batch scheduling

Data loads can even be scheduled hourly

Process Chains for modeling complex load processes using InfoPackagesand DTPs

Scheduling options for Process Chains using SAP NetWeaver batch

scheduling or Redwood Chronacle (OEM)


9/39 SAP AG 2006, HTWG AK SWT 101106 / 9

SAP NW2004s BI: Data Flow Concept Example

TRANSFORMATION

DataSource (PSA)

Source System

Process Chain

SAP NetWeaver BI

Data Transfer Process

InfoPackage

InfoProvider


10/39 SAP AG 2006, HTWG AK SWT 101106 / 10

Data Acquisition LayerData Source/Source Systems

Relational

Source

Multi-

Dimensional

Source

File XMLSAP

Source

DataSource

UD

Connect

DB

Connect

Web

Service

File

Interface

BI Service

API

e.g. MSS,

DB2, Teradata

e.g. Hyperion e.g. SAP ERP

Create

DB Connection and

DataSource
http://localhost/var/www/apps/conversion/tmp/scratch_7/01_Modelling_DBConnect_DS_u1.sim


11/39 SAP AG 2006, HTWG AK SWT 101106 / 11

New BI DataSource concept with NetWeaver 2004s

Highlights

unique look and feel for all of the DataSource Types

direct/remote access is always an option

preview feature is standard

automated coversions (e.g. date format detection)

InfoPackages only write into PSA

new Transformation handling

new Data Transfer Process logic


12/39 SAP AG 2006, HTWG AK SWT 101106 / 12

Source System Tree

Source Sytems categorized:

SAP vs. non SAP File vs. database

Relational vs.

Multidimensional DB

ABAP vs. Java

XML vs. Text/Binary


13/39 SAP AG 2006, HTWG AK SWT 101106 / 13

SAP NW2004s BI: Data Flow Concept Example

TRANSFORMATION

DataSource (PSA)

Source System 1

Process Chain

SAP NetWeaver BI

Data Transfer Process

InfoPackage

InfoProvider


14/39 SAP AG 2006, HTWG AK SWT 101106 / 14

Transformation Graphical UI

Source

fields

Target

fields

Rules

per

group

Note: Key figures, characteristics

and date fields are shown on the

same level (transformation group)


15/39 SAP AG 2006, HTWG AK SWT 101106 / 15

Transformation - Rules

Information on

Rule type

Currency/

Unit

Conversion

Source

fields

Target

fields


16/39 SAP AG 2006, HTWG AK SWT 101106 / 16

Data Transfer Process: Complex Example

DataStore Object 1

DTP

DataStore Object 2

DataStore Object 3

DTPDTP

DTPDTP

DTP

IP IP

TR

TR TR

TR

TR

DataSource (PSA) DataSource (PSA)

Source System 1 Source System 2

Process Chain

Process Chain

SAP Netweaver BIInfoProvider


17/39

SAP AG 2006, HTWG AK SWT 101106 / 17

Complex Implementation Example


18/39




Direct Access


Outlook and Summary


19/39

SAP AG 2006, HTWG AK SWT 101106 / 19

Two given Options for real-time Data Warehousing



DataSource Transformation Data Transfer Process (DTP)Flavours of ETL


Realtime Data Acquisition (RDA)

Integrated in data flow concept DataSources with specific Adapter Types like WebService Push and Real-time

Extraction from SAP System

DTP Type Real-time Data Acquisition

Daemon based processing with modified Request Handling

Alternatives

Direct Access

Modeling based on VirtualProvider

DataSources are Direct Access enabled

DTP Type for Direct Access

Driven by Query handling


20/39




Direct Access


Outlook and Summary


21/39

SAP AG 2006, HTWG AK SWT 101106 / 21

Direct Access: Simple Example

TRANSFORMATION

DataSource

for direct access

SourceSystem

BI

DataTransfer Processfor

Direct Access

VirtualProvider

Query


22/39

SAP AG 2006, HTWG AK SWT 101106 / 22

Direct Access - Implementation

Create VirtualProvider

Create

Transformation and DTP
http://localhost/var/www/apps/conversion/tmp/scratch_7/11_Modelling_DA_Transformation.simhttp://localhost/var/www/apps/conversion/tmp/scratch_7/10_Modeling_DA_Virtual_IP.sim


23/39

SAP AG 2006, HTWG AK SWT 101106 / 23

Direct Access: Complex Example

DTP forDirect Access

TR

Transactional Data

BIVirtualProvider

Query

DataSource

(direct access)

TR

Master Data

DataSource

(direct access)

Source System

Remote Characteristic

as InfoProvider

DTP for

Direct Access

O


24/39

SAP AG 2006, HTWG AK SWT 101106 / 24

Implementation - Overview

Create

Remote Characteristic as

InfoProvider

Verifymaster data

handling during

Reporting

Fi l Q R lt ith R t M t D t T t
http://localhost/var/www/apps/conversion/tmp/scratch_7/21_Verifiying_Rem_Master_Data_accessability.simhttp://localhost/var/www/apps/conversion/tmp/scratch_7/20_Modelling_DA_RemMD_IO.sim


25/39

SAP AG 2006, HTWG AK SWT 101106 / 25

Final Query Result with Remote Master Data Texts

Remote

Customer

Names

Customer names

from external

source

Di t A S i


26/39

SAP AG 2006, HTWG AK SWT 101106 / 26

Direct Access Scenarios

BI DataSource Types with Direct Access capabilities

File DataSource

Access file system of application server or local workstation

DB Connect DataSource

Access SAP NetWeaver DB platforms using ODBC technique

UD Connect DataSource

Based on the Universal Data Integration (UDI) concept

Access to any source system using SAP J2EE Connection Framework based on BIJava Connectors

JDBC Connectorfor any DB providing JDBC (MS Access, Teradata, )

ODBO Connectorfor any multi dimensional providing OLE DB for OLAP (Hyperion, )

XML/A Connectorfor sources providing XML for Analysis (SAP NW BI, )

BI Service API DataSources

Access to SAP ERP source systems


27/39




Direct Access


Outlook and Summary

St t i T ti l D i i M ki


28/39

SAP AG 2006, HTWG AK SWT 101106 / 28

Strategic vs. Tactical Decision-Making

Standard Data

Acquisition

Real-time Data

Acquisition

Goal

Strategic decision-

making

(long term planning)

Tactical decision-making

(day-to-day decisions)

Data StagingRequest oriented

(nightly batch job)

Stream oriented

(always active batch job)

Upload frequency 1/day 1/week 1/min 1/hour

Availability for reporting After a certain period Close to real-time

Computing PowerUsually done at night

(load balancing)Permanent resource

consumption

Use Real-time Data Acquisition only if necessary.

E l RDA i SAP ERP S i API


29/39

SAP AG 2006, HTWG AK SWT 101106 / 29

Example: RDA using SAP ERP Service API

SAP Source System

Service API

ApplicationDelta Queue

Real-time

Update

SAP NetWeaver BI

DataSource

Operational Data

Store

Daemon

Pull

~ 1/min

InfoPackage for Real-time

Data Acquisition

Data Transfer Process for

Real-time Data Acquisition

PSA

DataStore Objects

Service API-based real-time data

acquisition is a two-stageprocess:

Data is pulled into PSA by Real-

time InfoPackage

Data is transferred to DataStore

Object using Real-time DTP

In both cases the process of data

movement is initiated by the

system daemon

Real time Data Acq isition (RDA) Processing


30/39

SAP AG 2006, HTWG AK SWT 101106 / 30

Real-time Data Acquisition (RDA) - Processing

Daemon

Originally stands for Disk And Execution MONitor in the UNIX world

System process to initiate data loads at regular intervals: from one

minute to hourly

The BI Daemon data load includes three steps:

Initiate BI Service-API data pull using InfoPackage for RDA into PSA (SAP

source systems)

Track status of data transfer from source system

Initiate update of DataStore Object using DTP

Successful execution of each step is tracked in a control table

Allows restarting

Restart can be initiated so it starts at the next step after the last successfully

executed step

Daemon Monitoring


31/39

SAP AG 2006, HTWG AK SWT 101106 / 31

Daemon Monitoring

How to check the status of RDA daemon? Call transaction RSRDA or press Refresh in RSRDA

Is the daemon (batch job) still running?

The status of the daemon is indicated by one of the following icons:

Daemon active and running

Daemon not active

Daemon stopped with an error

Daemon stopped by user (will change to not active or erroneous)

Display runtime information about the daemon (context menu in RSRDA)

RDA using SAP ERP Service API


32/39

SAP AG 2006, HTWG AK SWT 101106 / 32

RDA using SAP ERP Service API

TRANSFORMATION

DataSource (PSA)

SAP ECC

Daemon Process

SAP NetWeaver BI

RDA-Data Transfer Process

RDA-InfoPackage

DS-Object

RDA using SAP ERP Service API Modelling View


33/39

SAP AG 2006, HTWG AK SWT 101106 / 33

RDA using SAP ERP Service API - Modelling View

Create

InfoPackage and DTP for RDA

RDA Example Using Service API Administration View
http://localhost/var/www/apps/conversion/tmp/scratch_7/40_SAPI_RDA_Mod.sim


34/39

SAP AG 2006, HTWG AK SWT 101106 / 34

RDA Example Using Service API Administration View

Create

a Daemon in the RDA Monitor

generating an open request waitingfor transactional SAP ERP PO data

Run SAP ERP Purchasing to push

a data record into BI using this

chanel

Monitor: Real-time Data Acquisition

RDA Implementation Scenario
http://localhost/var/www/apps/conversion/tmp/scratch_7/42_SAPI_RDA_PO_u1.simhttp://localhost/var/www/apps/conversion/tmp/scratch_7/41_SAPI_RDA_Admin_u1.sim


35/39

SAP AG 2006, HTWG AK SWT 101106 / 35

RDA Implementation Scenario

DataSource

PSA

DataStoreObject

DataStoreObject

OLTP

BI

Delta Queue

2LIS_02_VAITM Application

real-time

Update

Daemon

RDA

RDA

Enhance established Data Flow with RDA

capabilities

Implement additional DataStore Object for

operational reporting

Replace standard delta InfoPackage by RDA

InfoPackage

Regular data loads can be scheduled after

closing the RDA InfoPackage Request using

appropriate Process Chain feature Typically data is deleted regularly from the

DataStore Object supplied using RDA

Standard reporting can be enhanced by

operational reporting using the report-report

interface


36/39




Direct Access


Outlook and Summary

Real-time Data Integration with SAP NW BI


37/39

SAP AG 2006, HTWG AK SWT 101106 / 37

Real-time Data Integration with SAP NW BI

BI

(OLAP)

ERP(OLTP)

Abbildung von Prozessen


HR

Analyse von Prozessen

FI/CO

Einkauf, Fertigung, Vertrieb HR

Ziel

Entscheidungsuntersttzung (DS)

Planung

ERM(RDBMS)

MDM(M/R-DBMS)

Stamm- und Prozessdaten

Replikation

periodisch SAP NW BI ETL

SAP NW BI RDA

stetig

EAI

(SAP NW Process Integration)

sonstig SAP NW BI Direct Access

SAP

Terminologie

Complete

SAP Solution

Real-Time Data Warehousing with SAP NetWeaver BI


38/39

SAP AG 2006, HTWG AK SWT 101106 / 38

Real-Time Data Warehousing with SAP NetWeaver BI

Keypoints to take home

Rich standard ETL capabilities

Direct Access

allows querying data without data replication

any source system type (e.g. JDBC)

Real-time Data Acquisition

reduces data latency in the operational data store to ~ 1 min

Daemon based data loading


39/39

THANK YOU FOR YOUR

ATTENTION !

QUESTIONS SUGGESTIONS DISCUSSION

real-time data warehousing with sap netweaver business intelligence

Documents