itaipu a business activity monitoring system designed for business users prof. jacob slonim june 27,...

ItaipuA Business Activity Monitoring system designed for business users

Prof. Jacob Slonim June 27, 2009

Student: Azza Abouzied, MCSc

Motivation Related work Itaipu Usability Future workStream Cubes

Largest hydroelectric dam in the world, provides 20% of Brazil’s and 94% of Paraguay’s power

Outline

• Motivation

• Related Work

• Research Questions

• Our vision of BAM: Itaipu

• Multi-dimensional analysis

using Stream Cubes

• Usability

• Future Work

• Provide just-in-time analysis of business events– Business Intelligence (BI)

• Business Objects– Initiated research– Different approaches


Research Goal


• Enterprise’s Global Responsiveness and Agility

• Wal-Mart sales employees– Oil prices increase– Weather predictions– Canadian tire discounts garden products– Will we make the quarter?

Motivation


Business Activity Monitoring (BAM)

• Continuous queries over data streams– Continuous query

• has implications on multiple query optimization– Infinite, unbounded, data streams

• Single-pass processing, processing bounds, or approximation techniques required

• Ad-hoc queries

• Dynamic environment

• Target business end-users NOT technical analysts

• Complex Event Processing Systems– Cayuga– Origin: Business Process Management– Business Objects Approach

• Data Stream Management Systems– STREAM– Aurora, Borealis– TelegraphCQ– Origin: Monitoring sensor networks


Related Work


Research Questions1. Could we reuse DSMS for a BAM system?

2. Could we provide multi-dimensional analysis of data streams?

3. Could we simplify and expedite query write-up for business users?


Largest hydroelectric dam in the world, provides 20% of Brazil’s and 94% of Paraguay’s power

Itaipu• Spring

– Data Source

• Dam– Query processing

unit

• Delta– Results display and

storage


Itaipu’s big picture


• Brief design overview of Itaipu

• Two more questions left:

– Multidimensional data analysis of stream data

– Enhancing user-experience or usability

Recap & What’s Next?

city, sub_category

city, categorycountry, sub_category

country, category

country, allall, category

all, all

city, allall, sub-category

Cuboid

Base cuboid

Apex cuboid


Cube: a multi-dimensional data model

• Dimensions– Region: (city < country

< all)– Product: (sub-category

< category < all)

• Measures– Sum, count,

average

• SizeProduct

dimensionRegion dimension


Stream Cubes

• Han et. al introduced stream cubes

• Minimally-interesting (m-layer) and observation layers (o-layer)

• Partial Materialization– Popular drilling path


Conceptual representation of a tilted time frame

Stream Cubes

• Tilted Time frame– Time dimension– Distant aggregates

provided at larger granularities in comparison to more recent ones


Difference in perspective

• Popular drilling path– Users interested in o-layer and occasionally drill down– Path could be selected by an expert– Users drilling path is not likely to change

• Dynamic materialization path– User requests are distributed at different abstraction

layers– User requests are likely to change or evolve over the

course of the cube’s lifetime in the system

city, sub_category


country, category

country, allall, category

all, all

city, allall, sub-category


Drivers for change• External factors

– Oil prices increase– Export and oil-based products

affected– Shift towards viewing more

product information, less regional detail

• Competition• Seasonal• Market or economic


Dynamic Materialization Path

• Simple and efficient approach– Linear cost-effectiveness function– Dynamic programming approach

• Cost-effectiveness function– Memory cost– Popularity effect– Update potential

• Adapts to different operating environments– Weighted cost-effectiveness =- α(Memory cost) + β(Popularity) + γ(update potential)


Update potential?


α = 0.5,

β = 1.0 ,

γ = 0.75

Simulated example


city, sub_category


city, allcountry, category

country, all

all, all

all, category

all, sub_category

Resetting Path


Resetting paths on demand


Benefit of dynamic path materialization?

• Flexible– Follows user requests

• Low reliance on IT staff– Suitable for just-in-time operation

• How does it affect data stream processing?– Queue Size measure

• How does it affect memory consumption– Memory usage


Data rate ≈ 12,000

Experimental Setup


Experimental Setup


Results – Queue size


Results – Memory consumption


So, is it better?

• Conclusion– Situation dependent: more flexible solution– Not suitable when fixed memory guarantees are

required

• Are their better dynamic path materialization algorithms?– Probably– Our aim: simple, efficient, something that works!– Future improvements?


Recap & What’s Next?

• Brief design overview of Itaipu

• Optimization work with stream cubes

• Last question:– Enhancing user-experience or usability for Itaipu’s

business end-users


BAM user experience

• Bottleneck to the adoption of BAM– Operational decision making requires query

enabling technology

– The entire organization utilizes BAM: training effort needs to be minimal


A usability framework

Jarke’s & Vassiliou’s model:• Training Effort

• Repeated Efforts– Thinking

• Self-construction• Assisted-construction

– Input– Correction


Lessons Learned from database usability

• The simpler the task, the less query formulation effort is required.– Corollary: Limiting the functionality of a query

interface, makes it easier to use– Query by example– Query by forms

• Collaboration reduces the training effort– C-TORI– SearchTogether


Redrawn from a table in Jarke et Vassiliou, 1985

BAM’s user profile


Our usability solution

• Query Frames– Created by data analysts– Simplify query formulation to completing a query outline– Transformed easily to a form interface

• Query search– Each query or query frame is associated with textual

descriptions– Users and analysts could share queries and query frames– Passive collaboration– Keyword-based Google-like search interface


Query Frames


Dimension Query Frame


Itaipu’s Search Interface


Analytical Evaluation

Goal Query Frames Search

Training Thinking

Self construction Assisted construction

Input Correction


Hypothesis Result

A DSMS could be used as the core of a BAM system Dynamic path materialization is better suited for the data stream model Query entry could be simplified with the help of query frames and query sharing ≈

Final Recap


• Further studies to validate some our hypotheses

• A vision for BI

What’s in the future?


Conclusion

• Business Objects was convinced with our solution:– Itaipu’s architecture– Stream Cube extension– Query frame and query search

1 0

Thank for the support of:

NSERC, Business Objects,

Business Intelligent

Network

Questions ?

Spring


• Framework, based on data sources,– define

• Properties (allows drop)

• Data rate• Push/pull

– create:• Data Transformer• Data Streamer

• Data specification

Dam: Execution Engine


• Execution Engine– Variant of pipe &

filter architecture

– Similar to Fjords model

– Works like an Operating system

• Thread Manager• Queue Manager• Operator Builder

Execution Engine: Queue Manager


Execution Engine: Thread Manager


Execution Engine: Operator Builder


• XML multiple-query plan– Decouple query planning/optimizing from execution engine

• Java to XML Binding (JAXB)– Builds a content object model for each XML element

representing an operator

• Framework for building operators– Hierarchy of builders (Operator -> Aggregators -> SUM)– Update existing network of operators

Dam: Query Processing Unit


• Query Language– SQL-like with windowing constructs– Industry standard

• Current implementation– Extremely naïve– Open source tools

Delta


• End-user interface– Displaying and storing results– Searching for queries in the query catalog or inserting

new queries using SQL or modifying query frames

• Communicates with execution engine – Delta is thick client, execution engine is the server– XML messages over a TCP channel

• Memory cost

Example c1 = country, c2 = product category, |c1| = 5,

|c2| = 10, then |(c1, c2)| = 5 * 10 = 50

A good cuboid


A good cuboid


• Popularity

• Update Potential

Cost-effectiveness function


• Cuboid

• Path

itaipu a business activity monitoring system designed for business users prof. jacob slonim june 27,...

Documents

business usersprof

data streamscontinuous

data streamssingle

processing bounds

bam system

query writeup

city country allproduct

azza abouzied