performance und scalability - highqsoft€¦ · performance und scalability expectations and...

HighQSoft GmbH | www.highqsoft.de | 21.05.2015

News / Outlook / VisionsPerformance und Scalability

Ralf NörenbergDirector


2

Performance und ScalabilityTopics

1. Sketching Tomorrow2. Big Data Project A: „ODS as a Master“3. Big Data Project B: „ODS as a Slave“4. Expectations and Combination of Results


Physical Storage

Todays ODS Infrastructure and Architecture View

3

ODS Server

Application


Today’s ODS What are the questions?

4

Give me all measurements of vehicle “Ford”, type “Focus”, year “2015” and engine “3.0”.

“Meta data” result based on a “meta data” query.


Tomorrow’s ODS What are the questions?

5

Give me all <vehicles, sensors, …> of vehicle “Ford”, type “Focus”, year “2015” and engine “3.0”

“Meta data” result based on a combined “meta data & mass data” query.

where Fuel Consumption > 10 literswhere Speed Peak > 150 km/hwhere Avg. Temp > 35 °Cwhere Diesel Pipe Burning = yes


Tomorrow’s ODS Sketching the Future

6

Meta-Data

ODS Server

Applications„Big Data“ Query

(Correlations)

Query Interface

Reduction / Filtering of Information

Big Data Technology Adapter A

„Cache“ - Level

Mass-Data

„Cache“ - Level

Analysis Server

Routing of Information

Big Data Technology Adapter B


Tomorrow’s ODS Applications

7

Applications have the benefit of predefined questions:boundaries to limit possible questions / queries that can be askedStreamlining of information flow

provision of pre-cached result setsProvision of pre-configured information about mass-data


Tomorrow’s ODS Applications

8

Applications have the benefit of predefined questions:boundaries to limit possible questions / queries that can be askedStreamlining of information flow

provision of pre-cached result setsprovision of pre-configured information about mass-data


No Big Data required.

where Fuel Consumption > 10 liters = AVG ODSwhere Speed Peak > 150 km/h = MAX ODSwhere Avg. Temp > 35 °C = AVG ODSwhere Diesel Pipe Burning = yes = pre-configured ODS


Tomorrow’s ODS What are the questions?

9


We need new technologies (“big data”)!

where Fuel Consumption > 10 literswhere Speed Peak > 150 km/hwhere Avg. Temp > 35 °Cwhere Diesel Pipe Burning = yesin continuous time window of “30 min” within measurement


Tomorrow’s ODS Big Data Queries

10

Applications may but “correlation queries” surely require new technologies forAnalysis of mass dataReturn of result sets

For “random” queries, there are consequences on the solution architectureStreamlining of information flow is not possible

How many experts are using the system?How often are the same / likewise queries used?How will the provision of “cached” information work (for performance)?



11

Meta-Data

ODS Server


(Correlations)

Query Interface



„Cache“ - Level

Mass-Data

„Cache“ - Level

Analysis Server



Being “smart” is key! /Discussion currently focus on the very last part of the process

Customer Requirements /

Solution Specification

Customer Requirements /

Solution Specification

“open”



12

Meta-Data

ODS Server


(Correlations)

Query Interface



„Cache“ - Level

Mass-Data

„Cache“ - Level

Analysis Server



HQL Implementationavailable



13

Meta-Data

ODS Server


(Correlations)

Query Interface



„Cache“ - Level

Mass-Data

„Cache“ - Level

Analysis Server



Scalability available / Avalon Distributor



14

Meta-Data

ODS Server


(Correlations)

Query Interface



„Cache“ - Level

Mass-Data

„Cache“ - Level

Analysis Server



Merlin 2G ImplementationAvailable / Architecture Design analogous to Avalon Scalability (to be shown)



15

Meta-Data

ODS Server


(Correlations)

Query Interface



„Cache“ - Level

Mass-Data

„Cache“ - Level

Analysis Server



Big Data Research


16




17

Big Data Project A: „ODS as a Master“Project Facts

Project started April 2015 and is runningOur partner is in Ingolstadt (~10.000 employees Tier 1 supplier)

Why partner up?

a big partner is required for provision of scalable work environments

“ODS technology” and “customer IT infrastructure” are regarded to still be existent (now: Avalon and Oracle)

there is an existing challenge with measurements of big size (~50GB each)

there is an existing big data cluster running


18

Big Data Project A: „ODS as a Master“Objectives

1) Performance analysis of regular ODS system with big data use cases (bottlenecks?)

2) Identification of “BIG ODS” system designs and architectures

3) Performance analysis of “BIG ODS”

openMDM application data access

Avalon Server data management

Merlin Server integration / analysis

Frequent Updates within ASAM US Big Data Workshop

Working Prototype for ASAM Big Data Conference


19

Big Data Project A: „ODS as a Master“Planned Steps

1) Setting up a “SPARK”-Cluster and writing mass-data into it (partly done)

2) Retrieving channel data from cluster (driver development) (working on)

3) Performance Analysis considering “huge tables” / Solution Identification (to start soon)

4) Integration of measurement meta data into SPARK (purple elements) (planned)

Solve the challenge of “Joints” between Oracle and SPARK

5) Retrieval data out of BIG ODS system (planned)


20

Big Data Project A: „ODS as a Master“System Architecture

Spark Cloud / Cluster

Hadoop

Files

SQL

XY

1


21

Big Data Project A: „ODS as a Master“System Architecture


Files

Avalon Server

Driver

OracleJOB

Driver submits “orders” (to be generic)

Jobs accepts “orders” (specific to physical storage and ODS data model)

Master / Worker enable physical storage access

Master / Worker

2

3

4

4

5


22

Big Data Project A: „ODS as a Master“Sustantiated Big Data Statements


Files

Avalon Server

Driver

OracleJOB

Technology Boundary: ODS Server needs to run outside of cluster

Scalability: Multiple (local) clusters may run next to data generators and enable scalability

Confirmation: ODS Server / Oracle remain as structuring entities

Master / Worker

JOB


23




24

Big Data Project A: „ODS as a Slave“Project Facts

Project started April 2015 and is runningOur partner is

Why partner up?

a big partner is required for provision of a scalable middle-ware solution

Know-how of integration systems to middle-ware

Objectives

General Case Study and Prototype

Specific Project Realization


25

Big Data Project A: „ODS as a Slave“Today’s Setup

Third Party Analysis Tool

Avalon Server+ Physical StorageMoMa


26

Big Data Project A: „ODS as a Slave“Introduction of a Middleware


BUS - Middleware

Avalon Server+ Physical StorageMoMa

Log /Admin


27

Big Data Project A: „ODS as a Slave“Use-Case A: Connecting ODS and ODS


BUS - Middleware

Avalon Server+ Physical Storage


Combined ODSWeb


28

Big Data Project A: „ODS as a Slave“Use-Case B: Connecting ODS and None-ODS


BUS - Middleware


None-ODSThird Party Tool

None-ODS


Combined ODSWeb

29

Big Data Project A: „ODS as a Slave“Use-Case C1: Scalability of ODS Systems by Outsourcing (“Copy”)


Avalon Server+ Physical Storage None-ODSAvalon Server

+ Physical StorageAvalon Server

+ Physical Storage

BUS - Middleware



30

Big Data Project A: „ODS as a Slave“Use-Case C2: Scalability of ODS Systems by Outsourcing (“Move”)



BUS - Middleware


Tool XYThird Party Analysis Tool


Combined ODSWeb

31

Big Data Project A: „ODS as a Slave“Use-Case D: Connecting Locations


Avalon Server+ Physical Storage None-ODS

BUS - Middleware

Tool XY Tool XYThird Party Analysis Tool


Combined ODSWeb

32

Big Data Project A: „ODS as a Slave“Use-Case E: Scalability / Decentralization of indifferent Set-Ups


Avalon Server+ Physical Storage None-ODSAvalon Server

+ Physical StorageAvalon Server

+ Physical Storage

BUS - Middleware


Tool XY Tool XYThird Party Analysis Tool


33




34

Performance und ScalabilityExpectations and Combination of Results

ODS and Big Data will supplement each otherIf technologies like SPARK are sustainable, the actual physical storage is of secondary interest

Both project designs (master & slave) supplement each otherRealization of successful BIG ODS systems is realistic

This presentation shall encourage discussions throughout the day! Many topics of the overall picture will be discussed and presented today!

Our outlook is optimistic!

performance und scalability - highqsoft€¦ · performance und scalability expectations and...

Documents