bi to big data

8/10/2019 BI to big data

1/12

E-Guide

BI INTEGRATION

CHALLENGES:DATA ERRORS,

BIG DATA AND

REAL-TIME NEEDS


2/12

P A G E 2 O F 1 2 S P O N S O R E D B Y

Home

BI data integrationprocess challengedby real-time needs,big data

Data errors, othermissteps can waylayBI data integration-strategy

BI INTEGRATION CHALLENGES: DATA ERRORS, BIG DATA AND REAL-TIME NEEDS

SERDEMANDSFORfaster access to business

intelligence data and the rise of big data

analytics are complicating data integra-

tion for BI systems. This e-guide exam-ines both the drivers of BI data integration and key roadblocks,

including data quality and data loading issues. Find out how new

data integration techniques and tools can help organizations move

forward with integration projects.

U


3/12


Home




BI DATA INTEGRATION PROCESS CHALLENGED BYREAL-TIME NEEDS, 'BIG DATA'

Alan Earls

Business intelligence (BI) systems and their supporting data warehouses are

only as good as the data that goes into them. And if you arent properly handling

the BI data integration process, your end users -- and ultimately, your organiza-

tion -- may be in for trouble.

With BI tools becoming more and more pervasive in organizations, and

more critical to the success of business operations, making sure that you have

a well-designed and well-executed process for integrating BI data is of para-

mount importance, according to data management analysts such as Ted Fried-

man of Gartner Inc.

Friedman said Gartner sees data integration challenges related to BI as a

drag on the success of BI and analytics initiatives -- and a big reason for outright

project failures.

As the data that organizations are trying to harness gets more and more

complex, with more kinds and sources of data and now big data thrown into


4/12


Home




the mix, a significant amount of time and effort is involved in matching, clean-ing and preparing data for BI applications, he said. Its a darned hard problem,

particularly when you add in older, legacy systems where you sometimes need

to do archaeology first in order to interpret the data.

Another complicating factor is that things are changing in the world of data

integration technology as business users demand faster access to BI data.

ETL STILL BEST BET FOR BI DATA INTEGRATION?

The traditional workhorse technology for managing BI data integration is ex-

tract, transform and load (ETL) software that pulls data from source systems

in bulk batch processing jobs. Friedman said newer data integration techniques

offer lower latency than ETL tools do. For example, change data capture soft-

ware and other real-time data integration tools let you push new or modified

information to data warehouse and BI systems in real or near real time, which

can be particularly useful for tasks like fraud detection. It is streaming [data]

in granular form rather than big chunks in batch, which is what ETL is using,

he said.

Another option: federated and virtualized approaches to data integration

and delivery that dont move the data out of source systems at all but instead


5/12


Home




create consolidated views of data from multiple sources for BI uses. With datavirtualization tools, the integrated data doesnt persist anywhere, Friedman

said. Youre grabbing it in real time and joining it together and making it seem

as if it is one database somewhere to the applications using it.

Despite the emergence of this new wave of data integration and delivery

tools, though, Friedman thinks it would be a mistake to view ETL software as

obsolete or no longer valuable. ETL is still relevant, he said. We think there

will always be a role for ETL-style processing because not all data can or should

be delivered in real time.

Indeed, Friedman warned that data integration vendors are pushing sexy

real-time options for BI data integration when many organizations can still get

what they need from a batch approach. Real-time [integration] costs money

and it requires a change from what organizations have been doing, so there

needs to be a strong business case for it, he said.

ETL still has a role -- it is the heavy lifter of data integration, agreed Clau-

dia Imhoff, president of Intelligent Solutions Inc., a consultancy in Boulder,

Colo. Still, she noted that its newer competitors can be more flexible and faster

to deploy and are better suited to delivering timely data to business users for

operational BI applications.


6/12


Home

BI data integrationprocess challengedby real-time needs,

big data



REAL TIME NOT ALWAYS RIGHT BUT MORE OF A REALITYAlthough he acknowledges that real-time data integration for BI is frequently

neither necessary nor desirable, Barry Devlin, founder of 9sight Consulting

in Cape Town, South Africa, points out that BI and analytics applications are

increasingly moving in that direction. I think it is a really interesting time in

terms of how this will pan out, he said.

As an example of an experimental use case, Devlin cited the U.S. insurance

industry, where real-time data from cars -- braking and speed data, time spent

driving and other information -- is being transmitted to business users at in-

surance companies through mobile phone networks, enabling the insurers to

modify premiums or even provide rebates on the fly.

As Friedman noted, the increasing focus on capturing and analyzing big

data, including Web server logs, social media data and other forms of unstruc-

tured information, adds another layer of complexity to the BI data integration

process within many organizations.

James Kobielus, until recently an analyst at Forrester Research Inc., said

while he was still working there that unstructured data can be as critical as

structured data to what youre doing in BI and analytics. Even companies that

are still planning or just beginning to implement big data analytics programs


7/12


Home


big data



should look ahead and make sure theyre prepared for the data integration chal-lenges ahead, added Kobielus, who has since taken a job at IBM. You need to

be ready, he said, for things like massive data inputs from social media and

start to budget and staff up.

DATA ERRORS, OTHER MISSTEPS CAN WAYLAY BI DATAINTEGRATION STRATEGY

Data integration is a critical component of business intelligence (BI) processes,

which are predicated on pulling together data from multiple source systems

and consolidating it in data warehouses for analysis. But data management

analysts warned that a BI data integration strategy can quickly go awry if it isnt

well designed and properly executed.

For example, Claudia Imhoff, president of consultancy Intelligent Solu-

tions Inc. in Boulder, Colo., said bad timing can spoil everything: Data needs to

be loaded into a data warehouse in time to be used for the planned BI purposes.

Fully understanding an organizations BI requirements, particularly when


8/12


Home


big data



users need real- or near-real-time access to data, is a must for BI and data in-tegration teams, Imhoff said.

Good data quality is just as important, according to Imhoff, who said that

correcting and cleaning up bad data shouldnt be a function solely of the BI data

integration process. Errors are happening everywhere else along the way, so

you need to figure out where they are coming from, she said -- and then work

to prevent data mistakes from finding their way into source systems in the first

place. In effect, Imhoff added, data integration and BI professionals are given

the job of consolidating faulty data and then get the blame when it isnt perfect.

We need to get people to understand that they shouldnt just shoot the mes-

senger, she said.

Ted Friedman, an analyst at Gartner Inc. in Stamford, Conn., thinks that

not paying enough attention to data quality is the biggest BI data integration

danger companies face. Ive been following data integration for more than

10 years, he said. And I still spend days talking to organizations that are not

getting the usage and trust and acceptance and value out of their BI efforts

because the quality of the data is not good enough, and they havent done the

right things to fix that.

Data quality problems clearly affect more than BI data in wayward


9/12


Home


big data



organizations, Friedman said, but he sees poor data quality as one of the pri-mary barriers to successful BI programs. The shortcomings, he added, typically

result from not focusing [on data quality] early and often enough, and simply

not doing enough to mitigate quality issues as information is moved into data

warehouses.

James Kobielus, who was an analyst at Forrester Research Inc. in Cam-

bridge, Mass., before taking a job with a technology vendor earlier this year,

also pointed to missteps on data quality as a common source of trouble for BI

data integration efforts.

Organizations think they can simply load data from their various back-end

applications into a data warehouse and it will be usable without cleansing it or

doing match-and-merge or transform [processes], Kobielus said while he was

still at Forrester. But doing so sets up companies for some nasty surprises, he

added. For example, they end up with six records on the same person and dont

know which one is the right one, Kobielus said.

BI DATA INTEGRATIONS DRAMATIC EFFECT

Another big source of inconsistent data, and drama, stems from internal de-

bates over what constitutes a system of record, said Jill Dyche, co-founder of


10/12

P A G E 1 0 O F 1 2 S P O N S O R E D B Y

Home


big data



Baseline Consulting in Sherman Oaks, Calif. For example, she noted, there canbe arguments about which transaction system should be used as the source of

customer addresses. Such conversations often then turn to the definition of

address: Is it a customers billing address or shipping address, or its head-

quarters location if that differs from the other two?

Thats when the arguments ensue and business people become disaffected

with the BI teams ability to understand and deliver the right data, Dyche said.

So then someone just decides to forklift everything into a single database,

which the business people then refuse to use.

Barry Devlin, founder of 9sight Consulting in Cape Town, South Africa,

thinks the most problematic mistake is not including the right people in the

process of crafting a BI data integration strategy and plan. The people who

really understand data and what it means are a particular subset of the busi-

ness community who have been playing with data over the years -- they are

the gurus and the power users, Devlin said. As a result, he added, theyre best

equipped to define what data needs to be integrated in order to create effective

BI applications.

But in many cases, its left to the IT department to develop the data integra-

tion plan in addition to doing the implementation work, Devlin said. While IT


11/12


Home


big data



pros may have a reasonable understanding of an organizations data, to Devlinthey arent the real experts. Bringing the two groups together to work on BI data

integration can be a challenge, but its a must, he said.

Imhoff said there also is a strong tendency, especially in organizations that

are new to BI, to inadequately scope the data integration requirements of a

BI project and then put together an unrealistic schedule for delivering whats

needed. Integrating data and loading it into a data warehouse can take up as

much as 60% to 80% of the overall BI development effort, Imhoff said. And, she

cautioned, a project team that tries to do too much of that at one time can end

up falling flat on its face. You cant eat the whole elephant at once, she said.


12/12


Home


big data



FREE RESOURCES FOR TECHNOLOGY PROFESSIONALSTechTarget publishes targeted technology media that address

your need for information and resources for researching prod-

ucts, developing strategy and making cost-effective purchase

decisions. Our network of technology-specific Web sites gives

you access to industry experts, independent content and analy-

sis and the Webs largest library of vendor-provided white pa-

pers, webcasts, podcasts, videos, virtual trade shows, research

reports and more drawing on the rich R&D resources of technology providers to addressmarket trends, challenges and solutions. Our live events and virtual seminars give you ac-

cess to vendor neutral, expert commentary and advice on the issues and challenges you

face daily. Our social community IT Knowledge Exchange allows you to share real world

information in real time with peers and experts.

WHAT MAKES TECHTARGET UNIQUE?

TechTarget is squarely focused on the enterprise IT space. Our team of editors and net-work of industry experts provide the richest, most relevant content to IT professionals and

management. We leverage the immediacy of the Web, the networking and face-to-face op-

portunities of events and virtual events, and the ability to interact with peersall to create

compelling and actionable information for enterprise IT professionals across all industries

and markets.

bi to big data

Documents