bi isn't big data and big data isn't bi

74

Click here to load reader

Upload: mark-madsen

Post on 17-Aug-2015

6.475 views

Category:

Data & Analytics


0 download

TRANSCRIPT

Page 1: Bi isn't big data and big data isn't BI

SQL..

.

SQL!

SQL?

SQL Hadoop

BI Isn’t Big Data, Big Data Isn’t BI June, 2014 Mark Madsen www.ThirdNature.net @markmadsen

Page 2: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Summary

Common uses and commodity technology

lead to

Novel practices

lead to

Different data and different technology needs

lead to

New architectures

Lead to

Common uses and commodity technology

Page 3: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Our ideas about

information and

how it’s used are

outdated.

Page 4: Bi isn't big data and big data isn't BI

© Third Nature Inc.

How We Think of Users

Our design point is the passive consumer of information.

Proof: methodology

▪ IT role is requirements, design, build, deploy, administer

▪ User role is run reports

Self-serve BI is not like picking the right doughnut from a box.

Slide 4

Page 5: Bi isn't big data and big data isn't BI

© Third Nature Inc.

How We Think of Users

Our design point is the passive consumer of information.

Proof: methodology

▪ IT role is requirements, design, build, deploy, administer

▪ User role is run reports

Self-serve BI is not like picking the right doughnut from a box.

How We Want Users to Think of Us

Page 6: Bi isn't big data and big data isn't BI

© Third Nature Inc.

How We Think of Users What Users Really Think

Page 7: Bi isn't big data and big data isn't BI

© Third Nature Inc.

We think of BI as publishing, an old metaphor.

Publishing has value, but may not be actionable.

Page 8: Bi isn't big data and big data isn't BI

© Third Nature Inc.

When you first give people access to information that was unavailable…

OH GOD I can see into forever

Page 9: Bi isn't big data and big data isn't BI

© Third Nature Inc.

After a while it becomes the new normal

Page 10: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Planning data strategy means understanding the context of data use so we can build infrastructure

Monitor Analyze

Exceptions

Analyze

Causes Decide Act

No problem No idea Do nothing

We need to focus on what people do with information

as the primary task, not on the data or the technology.

Page 11: Bi isn't big data and big data isn't BI

© Third Nature Inc.

General model for organizational use of data

Monitor Analyze

Exceptions

Analyze

Causes Decide Act

No problem No idea Do nothing

Act within the process

Usually real-time to daily

Page 12: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Origin of BI and data warehouse concepts

The general concept of a separate architecture for BI has been around longer, but this paper by Devlin and Murphy is the first formal data warehouse architecture and definition published.

12

“An architecture for a business and

information system”, B. A. Devlin,

P. T. Murphy, IBM Systems Journal,

Vol.27, No. 1, (1988)

Slide 12 Copyright Third Nature, Inc.

Page 13: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Origins: in 1988 there was only big hair.

▪ No real commercial email, public internet barely started

▪ Storage state of the art: 100MB, cost $10,000/GB

▪ Oracle Applications v1 GL released; SAP goes public, enters US market

▪ Unix is mostly run by long-haired freaks

▪ Mobile was this

This is the context: scarcity of data, of system resources, of automated systems outside core financials, of money to pay for storage.

Page 14: Bi isn't big data and big data isn't BI

© Third Nature Inc.

General model for organizational use of data

Collect

new data

Monitor Analyze

Exceptions

Analyze

Causes Decide Act

No problem No idea Do nothing

Act on the process

Usually days/longer timeframe

Copyright Third Nature, Inc.

Page 15: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Straight line vs closed loop causality

BI is suited for stability, analytics for adaptation

Page 16: Bi isn't big data and big data isn't BI

© Third Nature Inc.

You need to be able to support both paths

Collect

new data

Monitor Analyze

Exceptions

Analyze

Causes Decide Act

Act on the process

Act within the process

Conventional BI, addition of EDM

Causal analysis, “data science”

Copyright Third Nature, Inc.

Page 17: Bi isn't big data and big data isn't BI

© Third Nature Inc.

The usage models for conventional BI

Collect

new data

Monitor Analyze

Exceptions

Analyze

Causes Decide Act

No problem No idea Do nothing

Act on the process

Usually days/longer timeframe

Act within the process

Usually real-time to daily

This is what we’ve been

doing with BI so far: static

reporting, dashboards,

ad-hoc query, OLAP

Copyright Third Nature, Inc.

Page 18: Bi isn't big data and big data isn't BI

© Third Nature Inc.

The usage models for analytics and “big data”

Collect

new data

Monitor Analyze

Exceptions

Analyze

Causes Decide Act

No problem No idea Do nothing

Act on the process

Usually days/longer timeframe

Act within the process

Usually real-time to daily

Analytics and big data is

focused on new use

cases: deeper analysis,

causes, prediction,

optimizing decisions

This isn’t ad-hoc,

reporting, or OLAP.

Copyright Third Nature, Inc.

Page 19: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Simple Complicated Complex

Assumption: Order Assumption: Unorder Assumption: Disorder

Cause and effect is repeatable & predictable

Cause and effect is separated in time & space, repeatable, learnable

Cause and effect is coherent in retrospect only, modelable but changing

Known Knowable Unpredictable

Standard processes, clear metrics, best practice

Analytical techniques to determine options, effects

Experiment to create possible options

Sense, categorize, respond Sense, analyze, respond Test, sense, respond

Like a bike Like a car Like economic systems

Copyright Third Nature, Inc.

Over time, processes are (usually) better understood,

so there is a movement of decisions from right to left,

where support is more automated.

Page 20: Bi isn't big data and big data isn't BI

© Third Nature Inc.

As practices evolve based on new capabilities…

A new level of complexity develops over top of the older, now better understood processes, leading to new data and analysis needs.

Page 21: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Technology captures observations. These change our understanding. New understanding changes practices.

Practices drive changes to technology, needing more data

Complexity will continue to increase

Page 22: Bi isn't big data and big data isn't BI

© Third Nature Inc.

I never said the

“E” in EDW meant

“everything”…

What do you

mean, “Just

doughnuts?”

Page 23: Bi isn't big data and big data isn't BI

© Third Nature Inc.

It’s going to get a lot worse

Not E

E

Conclusion: any methodology built on the premise that you must know and model all the data first is untenable

Page 24: Bi isn't big data and big data isn't BI

© Third Nature Inc.

“Big data is unprecedented.” - Anyone involved with big data in even the

most barely perceptible way

Page 25: Bi isn't big data and big data isn't BI

© Third Nature Inc.

We’ve been here before

Source: Bill Schmarzo, EMC

Page 26: Bi isn't big data and big data isn't BI

© Third Nature Inc.

“Big” is well supported by databases now

So

urc

e: N

ou

me

na

l, I

nc.

Page 27: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Orders of magnitude: 20 years ago TB, today PB

Shifts in data availability by orders of magnitude necessitate new means of managing and using it.

Page 28: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Analytics embiggens the data volume problem

Many of the processing problems are O(n2) or worse, so moderate data can be a problem for DB-based platforms

Page 29: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Much of the big data value comes from analytics

BI is a retrieval problem, not a computational problem.

Five basic things you can do with analytics

▪Prediction – what is most likely to happen?

▪Estimation – what’s the future value of a variable?

▪Description – what relationships exist in the data?

▪ Simulation – what could happen?

▪Prescription – what should you do?

Slide 29 Copyright Third Nature, Inc.

Copyright Third Nature, Inc.

Page 30: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Most people do not need special technology N

um

ber

of

peo

ple

The distribution of data

size is about normal, yet

these guys set the tone of

the market today.

Bigness of data

Copyright Third Nature, Inc.

Page 31: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Analytics: This is really raw data under storage N

um

ber

of

job

s

Microsoft study of 174,000 analytic

jobs in their cluster: median size ???

Bigness of data

Copyright Third Nature, Inc.

Page 32: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Working data for analytics most often not big N

um

ber

of

job

s

14 GB

Smallness of data

Copyright Third Nature, Inc.

Page 33: Bi isn't big data and big data isn't BI

© Third Nature Inc.

A Simple Division of the Problem Space C

om

pu

tation

Little

Lots

Data volume Little Lots

Big analytics, little data

Specialized computing,

modeling problems:

supercomputing, GPUs

Big analytics, big data

Complex math over large

data volumes requires

shared nothing architectures

Little analytics, little data

The entry point; SAS, SMP

databases, even OLAP

can work

Little analytics, big data

The BI/DW space, for the

most part

Page 34: Bi isn't big data and big data isn't BI

© Third Nature Inc. © Third Nature Inc.

What makes the actual data “big”?

Very large amounts

Hierarchical structures

Nested structures

Encoded values

Non-standard (for a database) types

Deep structure

Human authored text

“big” is better off being defined as “complex” or “hard to manage”

Copyright Third Nature, Inc.

Page 35: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Three kinds of measurement data we collect

The convenient data is transactional data.

▪ Goes in the DW and is used, even if it isn’t the right measurement.

The inconvenient data is observational data.

▪ It’s not neat, clean, or designed into most systems of operation.

The difficult and misleading data is declarative data.

▪ What people say and what they do require ground truth.

We need to make use of all three.

Copyright Third Nature, Inc.

Page 36: Bi isn't big data and big data isn't BI

© Third Nature Inc.

“big data” vs. transactions

The classic example of “structured data”

Transaction data includes:

▪ quantification details (date, value, count)

▪ reference data for explanation (product, customer, account)

▪ Lots of meaningful information

Reference data is usually shared across the organization, hence its importance. There are two parts:

▪ identifier to uniquely identify the subject

▪ descriptive attributes with common or standardized value domains

Transaction details

Reference data

Page 37: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Today it’s different data: observations, not transactions

Sensor data doesn’t fit well with current methods of collection and

storage, or with the technology to process and analyze it.

Copyright Third Nature, Inc.

Page 38: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Big data as a type of data: Transactions vs. Events

Transactions:

▪ Each one is valuable

▪ Mutable

▪ The elements of a transaction can be aggregated easily

▪ A set of transactions does not usually have important ordering or dependency

Events:

▪ A single event often has no value, e.g. what is the value of one click in a series? Some events are extremely valuable, but this is only detectable within the context of other events.

▪ Elements of events are often not easily aggregated

▪ A set of events usually has a natural order and dependencies

▪ Immutable

Page 39: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Example “big data”: Web tracking data USER_ID 301212631165031

SESSION_ID 590387153892659

VISIT_DATE 1/10/2010 0:00

SESSION_START_DATE 1:41:44 AM

PAGE_VIEW_DATE 1/10/2010 9:59

DESTINATION_URL

https://www.phisherking.com/gifts/store/LogonForm?mmc=link-src-email-_-m100109-_-44IOJ1-_-shop&langId=-1&storeId=1055&URL=BECGiftListItemDisplay

REFERRAL_NAME Google.com

REFERRAL_URL

http://www.google.com/search?sourceid=navclient&aq=0h&oq=Italian&ie=UTF8&rlz=1T4ACGW_enUS386US387&q=italian+rose&fu=0&ifi=1&dtd=204&xpc=1KoLqh374s

PAGE_ID PROD_24259_CARD

REL_PRODUCTS PROD_24654_CARD, PROD_3648_FLOWERS

SITE_LOCATION_NAME VALENTINE'S DAY MICROSITE

SITE_LOCATION_ID SHOP-BY-HOLIDAY VALENTINES DAY

IP_ADDRESS 67.189.110.179

BROWSER_OS_NAME

MOZILLA/4.0 (COMPATIBLE; MSIE 7.0; AOL 9.0; WINDOWS NT 5.1; TRIDENT/4.0; GTB6; .NET CLR 1.1.4322)

Page 40: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Web tracking data has a nested structure USER_ID 301212631165031

SESSION_ID 590387153892659

VISIT_DATE 1/10/2010 0:00

SESSION_START_DATE 1:41:44 AM

PAGE_VIEW_DATE 1/10/2010 9:59

DESTINATION_URL

https://www.phisherking.com/gifts/store/LogonForm?mmc=link-src-email-_-m100109-_-44IOJ1-_-shop&langId=-1&storeId=1055&URL=BECGiftListItemDisplay

REFERRAL_NAME Direct

REFERRAL_URL -

PAGE_ID PROD_24259_CARD

REL_PRODUCTS PROD_24654_CARD, PROD_3648_FLOWERS

SITE_LOCATION_NAME VALENTINE'S DAY MICROSITE

SITE_LOCATION_ID SHOP-BY-HOLIDAY VALENTINES DAY

IP_ADDRESS 67.189.110.179

BROWSER_OS_NAME

MOZILLA/4.0 (COMPATIBLE; MSIE 7.0; AOL 9.0; WINDOWS NT 5.1; TRIDENT/4.0; GTB6; .NET CLR 1.1.4322)

“unstructured” data

embedded in the

logged message:

complex strings

Page 41: Bi isn't big data and big data isn't BI

© Third Nature Inc.

The missing ingredient from most big data

Page 42: Bi isn't big data and big data isn't BI

© Third Nature Inc.

The creation and flow of data is different for observations, interactions and declarations

Data entry Extract Cleanse Load Use

Data

Generation

Store

Store

Use

Use

The process for most human-entered data

The process for much machine-generated data

Cleanse

Copyright Third Nature, Inc.

Page 43: Bi isn't big data and big data isn't BI

We collect large volumes of text, a rare practice ten years ago. Today we can turn text into data.

Categories,

taxonomies

Topics, genres,

relationships,

abstracts

Sentiment, tone,

opinion

Words & counts,

keywords, tags

Entities

people, places,

things, events, IDs Copyright Third Nature, Inc.

Page 44: Bi isn't big data and big data isn't BI

© Third Nature Inc.

You can store this data in an RDBMS, but…

Page 45: Bi isn't big data and big data isn't BI

Example data: Twitter Message API Payload

Looks like:

This is really just a record format

much like a DB row.

Datetime, userID, name, location,

description, message, message

metadata, etc.

But it’s In json or xml.

Page 46: Bi isn't big data and big data isn't BI

© Third Nature Inc.

@markmadsen Check out: From #MongoDB to #Cassandra: Why The Atlas Platform Is Migrating http://owl.li/cvxFK

A tweet has lots of fields, but one important one

The payload is free text but has other elements:

From these things you likely want to generate or link to reference data.

‘To’ username Hashtag Hashtag URL

Page 47: Bi isn't big data and big data isn't BI

© Third Nature Inc. © Third Nature Inc.

Internal payload elements form a new graph

The @elements point to other records and create a deeply linked structure.

You have to assemble the linked structure to see what’s really there, which means repeated scanning some/all of the data.

The derived pattern is interesting data, sometimes more than the individual messages.

Page 48: Bi isn't big data and big data isn't BI

© Third Nature Inc. © Third Nature Inc.

There are many patterns in the data

Follower / following networks are easy – they are explicit and independent of the events.

Community detection requires looking at patterns of @ communication in addition to follow relationships.

What do you do with these after discovery? Follower network Conversational communities

Page 49: Bi isn't big data and big data isn't BI

© Third Nature Inc.

More data: patterns emerge from lots of event data

Patterns emerge from the underlying structure of the entire dataset.

The patterns are more interesting than sums and counts of the events.

Web paths: clicks in a session as network node traversal.

Email: traffic analysis producing a network

The event stream is a source for analysis, generating

another set of data that is the source for different analysis.

Page 50: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Big changes for data warehousing workloads

The results of analytic processing can, often do, feed back into the system from which they originate.

Much of the data is being read, written and processed in real time.

Our design point was not changing tables and ephemeral patterns.

Page 51: Bi isn't big data and big data isn't BI

Unstructured is Not Really Unstructured

Slide 51

Unstructured data isn’t really unstructured: language has structure. Text can contain traditional structured data elements. The problem is that the content is unmodeled.

Page 52: Bi isn't big data and big data isn't BI

© Third Nature Inc. Slide 52

THE BIG CHANGE ISN’T TECHNOLOGY, IT’S ARCHITECTURE

Page 53: Bi isn't big data and big data isn't BI

© Third Nature Inc.

There are really three workloads to consider, not two

1. Operational: OLTP systems

2. Analytic: OLAP systems

3. Scientific: Computational systems

Unit of focus:

1. Transaction

2. Query

3. Computation

Different problems require different platforms

Page 54: Bi isn't big data and big data isn't BI

© Third Nature Inc.

The geography has been redefined

The box we created:

• not any data, rigidly typed data

• not any form, tabular rows and columns of typed data

• not any latency, persist what the DB can keep up with

• not any process, only queries

The digital world was diminished to only what’s inside the box until we forgot the box was there.

Page 55: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Layered data architecture

The DW assumed a single flat model of data, DB in the center.

New technology enables new ways to organize data:

▪ Raw – straight from the source

▪ Enhanced –cleaned, standardized

▪ Integrated – modeled, augmented, ~semi-persistent

▪ Derived – analytic output, pattern based sets, ephemeral

Implies a new technology architecture and data modeling approaches.

Page 56: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Food supply chain: an analogy for data

Multiple contexts of use, differing quality levels

Page 57: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Data infrastructure is a platform ▪ Any data – structures, forms

▪ Any latency –in motion, at rest

▪ Any process – query, algorithm, transformation

▪ Any access – SQL, API, queue, file movement

Page 58: Bi isn't big data and big data isn't BI

© Third Nature Inc.

The evolution of BI is to a data platform, which means separating application from infrastructure.

Derived data

Raw data

Infrastructure layer:

Process and analyze

Store and manage

Application layer:

Deliver and use

The new model also encompasses data at rest and data in motion

Multiple access methods

Enhanced

data

Multiple ingest methods

BI, data extracts, analytics, applications

The platform has to do more than serve queries; it has to be read-write.

Page 59: Bi isn't big data and big data isn't BI

© Third Nature Inc. © Third Nature Inc.

IT reality is multiple data stores and systems Separate, purpose-built databases and processing systems for

different types of data and query / computing workloads, plus

any access method, is the new norm for information delivery.

BI, Reporting, Dashboards, apps

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

1 Marge Inovera $150,000 Statistician

2 Anita Bath $120,000 Sewer inspector

3 Ivan Awfulitch $160,000 Dermatologist

4 Nadia Geddit $36,000 DBA

Data

Warehouse

Databases Documents Flat Files XML Queues ERP Applications

Source Environments

Data processing Stream

processing

Page 60: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Away from “one throat to choke”, back to best of breed

“The extremely specialized nature of mass production raises the costs of product change and therefore slows down innovation.”

- Abernathy, 1978

Tight coupling leads to slow changes.

In a rapidly evolving market componentized architectures, modularity and loose coupling are favorable over monolithic stacks, single-vendor architectures and tight coupling.

Page 61: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Staff and skills are a problem in a build market

@BigDataBorat: Give man Hadoop

cluster he gain insight for a day. Teach

man build Hadoop cluster he soon

leave for better job #bigdata

Page 62: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Technology Adoption

Some people can’t resist getting the next new thing because it’s new and new is always better.

Many IT organizations are like this, promoting a solution and hunting for the problem that matches it.

Better to ask “What is the problem for which this technology is the answer?”

Copyright Third Nature, Inc.

Page 63: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Four core capabilities big data technologies add

1. Unlimited scale of storage, processing

▪ Agility, faster turnaround for new data requests (but not a

replacement for BI)

▪ Fewer staff to accomplish same goals

2. New data accessibility

▪ More data retained for longer period

▪ Access to data unused due to cost or processing limits

▪ Any digital information becomes usable data

3. Scalable realtime processing

▪ Brings ability to monitor and act on data as events occur

4. Arbitrary analytics

▪ Faster analysis

▪ Deeper analysis

▪ More broadly accessible analytics

Page 64: Bi isn't big data and big data isn't BI

© Third Nature Inc.

An important Hadoop + cloud computing benefit

Scalability is free – if your task requires 10 units of work, you can decide when you want results:

10 servers, 1 unit of time

Cost is the same. Not true of the conventional IT model Time

1 server, 10 units of time

X X

Page 65: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Hadoop: a summary of the magic

1. Provides both storage and complex processing as part of the same platform

2. Makes parallel programming more accessible

3. Schemaless and file-based, therefore flexible

4. Inexpensive, reliable scale-out

5. Potential for fast, scalable ingest

6. Low-cost

Page 66: Bi isn't big data and big data isn't BI

© Third Nature Inc.

As a technology moves from emerging to commodity the nature of acquiring, using and managing it changes

Generate

options

Innovation

Novel practice

Maximize value

Maturation

Constrain

choices

Adaptation

Good practice

Optimize

Standardize /

minimize choice

Acquisition

Best practice

Minimize costs

Saturation Innovation

Copyright Third Nature, Inc.

Agile & open source* methods

6 Sigma & process methods

Page 67: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Today: repeating the experience of the 80s & 90s

This is the turbulent

phase of the market

as it goes through

rapid development,

then product and

service changes.

Copyright Third Nature, Inc.

The Internet combined with commodity computing is forcing a new

business and IT structural evolution, already underway.

Maturation Saturation Innovation

Page 68: Bi isn't big data and big data isn't BI

© Third Nature Inc.

How we develop best practices: survival bias

We don’t need best practices, we need worst failures. Copyright Third Nature, Inc.

Page 69: Bi isn't big data and big data isn't BI

© Third Nature Inc.

TANSTAAFL

Technologies are not perfect replacements for one another.

When replacing the old with the new (or ignoring the new over the old) you always make tradeoffs, and usually you won’t see them for a long time.

Page 70: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Questions? “When a new technology rolls over you, you're either part of

the steamroller or part of the road.” – Stewart Brand

Page 71: Bi isn't big data and big data isn't BI

© Third Nature Inc.

Welcome to the big data revolution, more of an evolution

Be pragmatic, not dogmatic

Page 72: Bi isn't big data and big data isn't BI

© Third Nature Inc.

CC Image Attributions

Thanks to the people who supplied the creative commons licensed images used in this presentation: acorn_blue.jpg - http://www.flickr.com/photos/rogersmith/314324893/ wheat_field.jpg - http://www.flickr.com/photos/ecstaticist/1120119742/ Phone dump - Richard Barnes

ponies in field.jpg - http://www.flickr.com/photos/bulle_de/352732514/

straw men.jpg - http://www.flickr.com/photos/robinellis/6034919721/

text composition - http://flickr.com/photos/candiedwomanire/60224567/ girl on cell tokyo .jpg - http://flickr.com/photos/8024992@N06/986538717/ hamadan people mosaic.jpg - http://flickr.com/photos/hamed/225868856/

twitter_network_bw.jpg - http://www.flickr.com/photos/dr/2048034334/ klein_bottle_red.jpg - http://flickr.com/photos/sveinhal/2081201200/ donuts_4_views.jpg - http://www.flickr.com/photos/le_hibou/76718773/ subway dc metro - http://flickr.com/photos/musaeum/509899161/

Page 73: Bi isn't big data and big data isn't BI

About the Presenter

Mark Madsen is president of Third Nature, a technology research and consulting firm focused on business intelligence, data integration and data management. Mark is an award-winning author, architect and CTO whose work has been featured in numerous industry publications. Over the past ten years Mark received awards for his work from the American Productivity & Quality Center, TDWI, and the Smithsonian Institute. He is an international speaker, a contributor to Forbes Online and on the O’Reilly Strata program committee. For more information or to contact Mark, follow @markmadsen on Twitter or visit http://ThirdNature.net

Page 74: Bi isn't big data and big data isn't BI

© Third Nature Inc.

About Third Nature

Third Nature is a research and consulting firm focused on new and

emerging technology and practices in analytics, business intelligence,

information strategy and data management. If your question is related to

data, analytics, information strategy and technology infrastructure then

you‘re at the right place.

Our goal is to help organizations solve problems using data. We offer

education, consulting and research services to support business and IT

organizations as well as technology vendors.

We fill the gap between what the industry analyst firms cover and what IT

needs. We specialize in product and technology analysis, so we look at

emerging technologies and markets, evaluating technology and hw it is

applied rather than vendor market positions.