business intelligence 2.0: a general overview · 2011. 7. 7. · technical challenges of the new bi...

Dept. Lenguajes y Sistemas Informáticos

Business Intelligence 2.0: a General Overview

Juan [email protected]

First European Business Intelligence Summer School (eBISS), Ec. Centrale, Paris, 2011Juan C. TrujilloAlejandro Maté

2

ContentIntroductionBI & Data Warehouses in a NutshellBasic Concepts related to BI 2.0Influence from the Web on BITechnical Challenges of the new BI 2.0General Overview of Tools Stepping Towards BI 2.0Conclusions


3



4

IntroductionThe use of Business Intelligence solutions has been steadily increasing

In the recession period, the BI market grew 4%

BI allows the business to gain a competitive edge by analyzing the data of the organization


5

IntroductionTraditional technologies to support BI processes range from Data warehousesto OLAP and Data mining.

These technologies allow to query the organization’s internal data

However, a new trend has emerged: analyzing data from outside the organization.


6

IntroductionFor example, including information like:

Retail prices of products sold by competitorsOpinions from customers…

Result: Richer analysis and better support for the decision-making process


7

IntroductionHowever this trend is bidirectional

As BI applications include information from the Web, These applications have also been evolving towards web technologies.


8

IntroductionEvolution driven by technologies appeared in the Web 2.0:

Social Networks (e.g. Facebook)Graph and linked dataInteractive Web applications Cloud computingCollaborative NetworksProcess IntelligenceSoftware as a Service…


IntroductionSome authors call it BI 2.0, others BI 3.0…

Which are the common aspects that define the new BI?

How is the web affecting BI and whichnew features are being included fromthis influence?

9


IntroductionWhich technical challenges must be overcomed?

Which are already solved, which requirefurther research?

Which features are being integrated by BI tools?

10


IntroductionA first look into a BI 2.0 architecture…

11


12



BI & Data Warehouses in a Nutshell

Traditionally, BI has focused onanalyzing huge amounts of data tosupport the decision-making process

Transactional databases are notadequate for this task:

Difficult to retrieve the necessaryinformationPerformance

13



Necessary to develop the decisional database with an alternative design:

Focus on the information being analyzedImproving the performance

Result: Vendors implement the logical models Star Schema, Snowflake, Fact Constellation,…

[Kimball, 96]

14



15

TimeTime

ProductProduct

StoreStore

Single ColumnSingle ColumnTime KeyTime Key

Single ColumnSingle ColumnProduct KeyProduct Key

Single ColumnSingle ColumnStore KeyStore KeyCompositeComposite

KeyKey



However, as the logical model isdesigned to store data and improveperformance:

It does not take into account analysts’needsSchema is still difficult to understandDifficult to retrieve information (SQL queriesover the database)

16



The logical level lacks detail to specifymultidimensional information

It is necessary to create models with a higher abstraction level: conceptual models

17



Different conceptual models have beenproposed to solve this issue within data-driven approaches[Abelló et al. 2006][Trujillo et al., 2001][Luján-Mora et al. 2006][Sapia et al. 2004][Tryfona et al. 1999] [Golfarelli et al. 99] and many more

Unfortunately, up now none has beenaccepted as a standard

Although supported by tools and checked in real-world projects

18



19

Fact

DimensionsLUCENTIA



Is the multidimensional information notrelevant?

No, BI tools have also implemented theirown representationsInformation related to facts, dimensions, and hierarchies is highly relevant for theanalysisMoreover, it enables to query the DW usingOLAP and MDX queries

20



Nevertheless, being able to query theDW does not mean that it satifies user’sneeds

Solution? Perform a requirementsanalysis stage to design the DW

Using a higher abstraction level tocommunicate with users

21



As previously, there is currently no standard for the RequirementsEngineering (RE) stage[Giorgini, Golfarelli, Rizzi, 2008][Mazón et al. 2007]

However, the RE stage allows us toidentify and guarantee that the analysts’needs are met

22



23

User’s Requirements Approach

LUCENTIA



A complete hybrid DW developmentapproach has been proposed in theLucentia Research Group[Trujillo et al., 2001] [Mazón et al. 2008][Mazón et al. 2009]

Considers both user requirements anddata sourcesHybrid approaches allow us to identifyproblems in early stages

24



25

Hybrid Approach by using MDA



26

Lucentia BI Tool



Since there is a proposal forautomatically deriving DWs, has everyproblem been solved in this area?

No. There are a series of challenges stillopen

Traceability of user’s requirementsQuality measures to drive the designSecurity constrainst from the early stagesAnd many more

27



Business processes evolve, therefore theDW must support this evolution

We require to analyze informationpresent in the Web

This information is typically unstructuredAccuracy and correctness are notguaranteed

28


29



Basic Concepts related to BI 2.0When talking about BI 2.0 it is importantto define some basic concepts

Real-time [Thiele et al. In Press], …

All the information used must be fresh andup-to-dateExceptional situations previously unknown

E.g. Sales data without the list of products

30


Basic Concepts related to BI 2.0Real-time information

31


Basic Concepts related to BI 2.0Software as a Service (SaaS)[Essaidi et al. In Press]

Software is now consumed as a remote service

Use of Service Oriented Architecture (SOA) and SOA Protocol (SOAP) for interoperability

Recently applied to BI solutions, resulting in BI as a service

32


Basic Concepts related to BI 2.0Cloud computing[Armbrust et al. 2009], [Larry Ellison, Wikipedia]

Integration of several, heterogeneouselements into a network

Middleware provides homogeneousinterface

Services provided consumed through SaaS

Supports the addition of new elements33


Basic Concepts related to BI 2.0Collective Intelligence[Gruber et al. 2008]

Originally refers to emerging behavioursE.g. Ant colonies can solve the TravellingSalesman Problem

Social Networks also present emergingbehaviours

Decentralized groups are able to take decisionsas a group and promote initiatives

34


Basic Concepts related to BI 2.0Crowdsourcing[Howe et al. 2009]

Delegating a task to a crowde.g. Mechanical Turk services in Amazon

Each individual contributes with a littleeffort to the global goal

Depending on how the crowd is organized, the collective intelligence can achieve bettersolutions than a single expert

35


Basic Concepts related to BI 2.0Social Networks[Berthold et al. In Press][Golfarelli et al. In Press]

Group of participants which can interactwith each other

Typically they collaborate, achieving goals fasterand with better results than a single individual would

The most relevant data are thecontributions from the participants and therelationships between them

36


Basic Concepts related to BI 2.0Linked Data[Berlanga et al. In Press] [Bizer et al. 2009]

Knowing the relationships between eachpiece of data and the restIn order to be able to reason and infereknowledge, the relationships must be semantically taggedAllows to obtain knowledge automatically

37


Basic Concepts related to BI 2.0Opinion mining[Balahur et al. In Press]

Describing the general feelings of a group of peopletowards a certain element

Requires to analyze unstructured data, understandits content and obtain a conclusion

Highly relevant to identify how customers perceiveproducts

38


Basic Concepts related to BI 2.0Process Oriented BI[Golfarelli et al. 2004]

Point of view focused on Business processesand their logic

Tries to relate the stored data to theprocess performance

Extensions of BPMN 2.0

Allows to identify and restructure businessprocesses presenting problems

39


40



Influence from the Web on BIAs society evolves, the ratio ofconnectivity has increased

The business environment is rapidlychanging

Physical barriers dissapearBusiness provide their services online

41


Influence from the Web on BICustomers can access a wide variety ofoffers with no additional cost

The customer becomes more critic

New technologies allow customers tointerchange opinions

Social Networks, Twitter, online reviews…Customers influence other customers

42


Influence from the Web on BIBusinesses need to consider as muchinformation as possible when takingdecisions

What do the competitors offer? What do our customers thinkabout our products?

Decisions must be agileOtherwise, customers will leave

43


Influence from the Web on BIHow is BI and BI tools being altered by this new dynamic environment?

New interfacesUsing always fresh, up-to-date informationDecisions are no longer isolatedData is presented in a more significativemannerFocus on analyzing the immediate future

44


Influence from the Web on BINew interfaces

New requirement introduced: data must be checked from anywhere

Web interfaces replace desktop applications

Introduction of Mobile BI

45


Influence from the Web on BIMobile BI

Accessing data using mobile devices

Limitations in screen size and memoryTransfer only the necessary dataShow only the necessary information

Navigation must be interactive and simple

46


Influence from the Web on BIUp-to-date periods

Traditionally information wasprovided in the form of reports

However, currently reports are only checkedto identify the source of an existing problem

They are not interactive, and difficult to relate tobusiness goalsThe information provided by reports arrives toolate

47


Influence from the Web on BIIs monthly or weekly information freshenough?

In order to answer this question let usconsider an example:

After the recent security breaches in Sony, how much time was required for its imageto be hurt in the whole world?

Under a day

48


Influence from the Web on BIDecisions are no longer isolated[Berthold et al. In Press][Golfarelli et al. In Press]

Traditionally, decisions would be taken by executives in an isolated manner

However, it has been proposed that it isbetter to take decisions using collectiveintelligence or even crowdsourcing

Often, employees have relevant knowledge regardinga specific problem

49


Influence from the Web on BITwo main alternatives for takingdecisions in group:

Discuss the decision through collaborativeBI with other employees (i.e. interacting likein a Social Network)

Allow employees to enrich the existing dataand contribute with their own information

Then, exploit this information in order to take a decision

50


Influence from the Web on BIIn order to achieve these collectivedecisions, data should be interactive

Users should be able to easily interchangeinformation

Did you know that Excel is the 3rd most used BI tool in Spain? [Penteo 2011]

Users should be able to make annotationsand enrich the data with relevantinformation

51


Influence from the Web on BIData is presented in a significant way

Traditionally, and currently by default, toolsfocus on how to present aggregated data

Bar graphs, spreadsheets, stacked bars…

However, decision-makers wish to use thedata to identify which strategies are working and which ones are not

52


Influence from the Web on BITendency:

Relate the data to balanced scorecards by means of dashboards

Allows us to easily identify the status of ourbusiness strategy

Other proposals relate data directly tobusiness goal models or to business processmodels

53


Influence from the Web on BINew focus of analysis

Long-term decisions are still importantHowever, due to the dynamic environmentthere is a new necessity to focus on theimmediate future

Apparition of problems which require immediateattention

Still more visualization is needed

54


Influence from the Web on BIIncreasing number of short-termdecisions

These decisions require real-time information

i.e. Do I need to restock my products to meetthe demand for the rest of the day?

Reliance on predictive data mining, with a strong time restriction

55


56



Technical Challenges of the newBI 2.0

The new aspects envisioned for BI 2.0 cannot be accomplished unless weovercome a series of technologicalchallenges

Some of these challenges have alreadybeen thoroughly studied while others are still open for further research

57



Making a brief review we require:Support for real-time dataScalable architecture for multiple collaborating usersaccessing the DWInclude Web data into the analysisProvide predictive algorithms to analyze the informationAnalyze and relate business processes to the stored dataEstablish semantic relationships between data in order toautomatically infere knowledge (linked data)

58



Real-time Data WarehousesReal-time Data Warehouses have been thefocus of research in the past years

The most significant difference betweentraditional and real-time DW is how data iscaptured

59



Real-time Data WarehousesTraditionally, data is captured in operationaldatabases when a transaction is processed

Then, at some point defined by therefreshing cycle, all the new data is loadedthrough ETL processes

This process is known as bulk-feed

60



Real-time Data WarehousesBulk-feed has several drawbacks:

Negative impact on performance of bothoperational and decisional databasesThe DW does not have the most up-to-date data

Solution?Alter the way of capturing dataObtain the information simultaneously as it isstored in transactional databases (trickle-feed)

61



Real-time Data WarehousesIn order to load transactions into the DW as theyare captured we can:

Use triggers in transactional databasesExtract the information from logsUse replication techniques

The ETL process is transformed into a modeledparallel flow of data towards the DW

Information may be incomplete at certain pointsImportant to model unexpected flows (exceptions)

62



Real-time Data WarehousesFinally, as queries must be processedsimultaneously as data is being loaded weneed to consider:

Minimizing query delay by using parallelizationor main-memory databasesUsing replicated tables, in order to guaranteethe correctness of the analysis

These tables can be swapped in short cycles

63



Real-time Data Warehouses

More in Dagsthul seminar: Real Time DWs64



ScalabilityBI 2.0 envisages constant real-time data flows into the DW

Number of users querying the DW is alsoexpected to increase

The scalability of the system becomes animportant factor

65



ScalabilityHow can we increase the scalability of theBI system?

How can we increase the scalability of a systemin general?

More powerful single pieces of hardwareMore hardware elements

How does this translate into BI?

66



ScalabilityFirst option: More powerful pieces of single hardware

In BI this means betterdedicated servers

Exadata (Oracle) TwinFin (Netezza)

Information is stored by the organizationPotentially more securePrivacy

67



ScalabilitySecond option: Simply more hardware

The recent apparition of cloud services allows usto create a flexible system

Use more power as you needPay-as-go

Two approaches:Public Clouds (i.e. Amazon, Azure Cloud, iCloud)Private Clouds (with your own high-end servers!)

68



Processing semi-structured andunstructured data

Highly relevant information is posted onlineCustomers’ opinionsRetail prices from competitorsReviews of products

However, this information is not structured

69




Two different situations:Semi-structured data (XML)

Presents some basic structureThe structure helps in interpreting and integrating thedata into the system

Unstructured dataNLPIs Hadoop the right solution ?

70




How to include the web information into theBI System?

Integrate this information into the enterprise DWCorrectness of the data inside the DW will not be guaranteed!!

Create a separate DW for Web informationLink this information as a detailed view aftergenerating the analysis cube

71



72



Predictive data mining algorithmsAs the focus of analysis shifts towards theimmediate future, the importance ofpredictive analysis increases

Descriptive techniques are pushed into thebackground

Historic information and classifications are stillimportant but not the focus

73



Predictive data mining algorithmsIt is necessary to predict what will happenin the immediate future

A number of existing predictive techniquesalready exist

However, they have not been designed tofulfill the current needs in BI 2.0

74



Predictive data mining algorithmsHuge amounts of data are required to takea decision

This situation requires automatic analysis

Strong time constraintsThe result must be provided to solve a short-term problemWe need an answer, even if it is not the best answer

75



Analyzing business processesBusiness processes are gaining increasedattention from the research community

By modeling business processes, we are able to identify the flow of data through thebusiness activity

76



Analyzing business processesAdvantages of modeling businessprocesses:

Easier to understand the business activityIdentification of deadlocksAllow us to perform Business ProcessIntelligence

Which steps in the process are not working as intendedWhich processes should be remodeled

77



Analyzing business processesChallenges:

Business process models lack information aboutthe structure of the underlying data

Necessary to relate the existing data withbusiness process models in order to analyzethem

78



Linking dataAs we have previously seen, the most importantfeature of a piece of data in BI, is its relationshipswith other pieces of data

Relationships allow us to reason and infere knowledge

There are different kinds of relationshipsSome are explicitly modeledSome are implicit in the data

79



Linking dataExplicit relationships:

Explicit relationships are those already modeledin the system

i.e. Facts and Dimensions

They are the basis for analysis and structuringthe information

80



Linking dataImplicit relationships:

Although not explicitly modeled, implicitcorrelations between data can be discovered

Data mining, artifical intelligence can help toidentify these relationships

However, they require that all the information isincluded and related in the analysis

81



Linking dataWhat about the analysis of different sets ofdata?

e.g. Decrease in sales in our products related toan increase in sales from our competitorsUnless these sets are not joined in a single analysis, this information will not be identifiedHowever, we cannot join all the information toperform a single analysis of the whole system

82



Linking dataNevertheless, if the important relationshipsare identified and modeled, new knowledgecan be inferred

Ontologies for modeling the domainPreserving the existing relationships usingtraceabilityDomain-dependent

It is important to include meaningfullsemantics for the analysis

83


84



General Overview of Tools Stepping Towards BI 2.0

How are BI vendor tools integrating BI 2.0 features?

Which features are better supported?

Which ones are most lacking?

85



Brief analysis of tools to provide anoverview

MicrostrategyPentahoCognosSAS MicrosoftSAP

86



MicrostrategyPositive aspects:

Web interface. Includes dedicated Mobile BI supportScorecards and Dashboards connecting KeyPerformance Indicators (KPI) to provide visibility

Limitations:Limited predictive analysis supportInteraction and collaboration between users?Integration of business processes?

87



PentahoPositive aspects:

Open source, multi-platform, Web interfaceIncludes Dashboards for presenting linked dataSome predictive algorithms includedIncludes some collaborative features when integrated withLifeRay

Limitations:Designing and integrating dashboards requires some effortInteractivity and data enrichment?Integration of business processes?

88



Example of a BI web interface (Pentaho)

89



Cognos (IBM)Positive aspects:

Web interface. Support for Mobile BIDashboards and ScorecardsCollaborative support and data enrichment withannotations

Limitations:Predictive analysis support?Business processes?

90



SAS:Positive aspects:

Web interface. Support for Mobile BIHighly customizable DashboardsAllows linking elementsSpecial visualization tools

Limitations:Predictive analysis?Collaborative BI?Business processes?

91



Microsoft BI:Integration of several tools:

ExcelPowerPivotSQL ServerSharepoint

92



Microsoft BI:Positive aspects:

Web interfaceDashboards and Scorecards through mashupsAllows to link elements and interact with otherusersSupports adding tags to our profile in sharepointAnalysis through Excel and PowerPivotAllows to include data from the Web

93



Microsoft BI:Limitations:

Additional effort to use different technologies at the same timePredictive analysis limited to Excel functionsLacks some collaborative functions

Direct interaction between usersAdding annotations to the data, so other BI users can see them

94



SAP:As in the previous case, composed ofvarious tools

Provides complete support for analyzing thebusiness strategy combining desktopapplications with web applications

95



SAP:Positive aspects:

Web interface for ad-hoc analysis. IncludesMobile BI supportDashboards and scorecardsComplete workbench for data mining tasksCompatibility with other vendors tools, empowering analysis capabilities

ExcelEnterprise applications

96



SAP:Limitations:

The collaboration between users is limitedLacks support for enriching dataInteraction between users is not integrated in thesystemThese aspects are being improved

97



Final considerations:Most tools support cloud computing using a SaaS or BI as service approach

Some tools include support for using the cloud in specific tasks

Cognos: Guide to deploy the systemMicrosoft: Deploying the system into Azure CloudPentaho: Data Integration in the cloud

98



Final considerations:Visibility of the business strategy is providedmainly by using Dashboards

Requires some effort to build the desireddashboard and it is not always intuitiveCould be improved by adding support forbuilding the dashboard in an interactive way

Collaborative aspects are limited

99


100



ConclusionsBI 2.0 has to deal with several aspects:

Real-time analysisIntuitive and interactive analysis fromanywhereCollaboration between decision-makersLinking and enriching dataFocusing on the immediate future…

101


ConclusionsFurther research needs to be done in:

Predictive algorithms with strong time restrictionsIdentify the most effective way ofpresenting the dataDevelop a series of best practices whentaking decisions in a collaborative mannerProcess Intelligence and Process Mining…

102


ConclusionsFurther research needs to be done in BI:

103

ConclusionsIn our Lucentia Research Group:

Process IntelligenceMining processesLinking processes to data from BPMN 2.0

Traceability of user’s requirementsBI 2.0 security and qualityWeb WarehousesAdvanced visualization techniquesMore:

http://www.lucentia.esRecent publications on DB&LP

104

Conclusions

105

Lucentia BI ToolTo be extended

+ User’s requirements for BI 2.0

+ PIM, PSM adapted + Automatic transformations adapted

+ Reverse engineering adapted to new BI 2.0 sources (e.g. docs,…)

+ New traces

Dept. Lenguajes y Sistemas Informáticos

THANK YOU. QUESTIONS?

Business Intelligence 2.0: a General Overview

Juan [email protected]

business intelligence 2.0: a general overview · 2011. 7. 7. · technical challenges of the new bi...

Documents