big data is only half the story - qlik & big data

Guidelines

www.ing-presentations.intranet

Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette

brandportal.qlikview.com

Tertiary accent colours

Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0

Primary brand colour

1

Big Data is

Only Half the Story

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


Big Data is Only Half the Story

#VizYourWorld

Mischa van Werkhoven

Principal Solution Architect

@MischaDaniel

Netherlands, 2015

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


3

The final part of the story is

adding context and relevance

and delivering it to people

at the point of decision.

“ “


Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


4

How many bytes

does it take to store

CGI effects for

Avatar?

Bytes

Petabytes

Megabytes

Gigabytes

Exabytes

Yottabytes

Kilobytes

Terabytes

Zettabytes

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


5

Bytes to store Avatar CGI effects

Bytes

Kilobytes

Megabytes

Gigabytes

Terabytes

Petabytes

Exabytes

Zettabytes

Yottabytes

Bytes

Kilobytes

Megabytes

Gigabytes

Terabytes

Petabytes Exabytes

Zettabytes

Yottabytes

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


6

Expanding on 3 fronts

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


7

Hype Cycle 2013

Big Data

Internet of Things

In-Memory Analytics

As of July 2014

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


8

Hype Cycle 2014

In-Memory Analytics

Big Data

Internet of

Things

As of July 2014

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


9

Data

It's Not About the Data:

People Make Decisions

Opinions Intuition

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


10

To Make Decisions

People Seek Answers to Questions

Data provides these answers

• Provide all relevant data

• Provide context

• At the point of decision

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


11

To Make Decisions

People Seek Opinions of Others

• Enable Social Collaboration

• At the point of decision

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


12

Intuition

People rely on their Intuition

to Make Decisions

Even though Google provided all

relevant route and traffic information

I chose the longer alternative…

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


13

It's Not About the Data:

People Make Decisions

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


14

Data Warehouse

• Improved Quality of decision making

• Speed Improvement?

• Technology driven

• Rely heavy on IT

Back in the days…

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


15

Data lake

• Increased context

• Quality of decision making?

• Speed improvement?

• Technology driven

• Rely heavy on IT

Nowadays…

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


16

The New IT: How Technology Leaders Are Enabling

Business Strategy in the Digital Age, Jill Dyché, 2015

You’ve been loading data

into a data warehouse for

as long as we can

remember. But no one

asked us if we needed any

of that data.

“ “

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


17

If Data Is The New Oil...

Shouldn't We Treat It That Way?

• Nobody Invests In Drilling At Random

• Data Lakes Are As Useless As An Oil Reserve

• After All That Effort, Fuel Is Not Even The Point

http://www.qlik.com/blog/posts/peggysue-werthessen/if-data-is-the-new-oil-shouldnt-we-treat-it-that-way

http://www.qlik.com/blog/posts/peggysue-werthessen/if-data-is-the-new-oil-shouldnt-we-treat-it-that-way

Qlik & Big Data

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


19

Integrate Data from Anywhere

to Add Context and Relevance

Improve Decision Making Quality

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


20

Visual Analytics at the Point of Decision

Reporting Guided

Self Service

Web Integration

Custom

Increase Speed of

Decision Making

Big Data

Deployment Models

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0

Primary brand colour Flexible Big Data deployment models

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


23

In-Memory Approach

• Loads compressed data into memory

• Enables associative search and analysis

• Supports 100’s millions to billions of rows of data

• MOLAP is most common implementation approach

In-Memory

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


24

Direct Discovery Approach

• Associative Data Discovery for Big Data

• Combines in-memory data with in-

database data

• Seamlessly combines Big Data with

other data sources

• Massive data scalability

• Easy to implement

• Rapid Big Data app development

• HOLAP implementation

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


25

Application Architectures In-Memory

Many Big Data use cases can be addressed purely in memory

Drill-to-Segmented-Detail

East Region App

(150 MM rows)

West Region App

(150 MM rows)

North Region App

(150 MM rows)

South Region App

(150 MM rows)

Summary Dashboard

(10 MM rows)

Drill-to-Detail

Summary Dashboard

(10 MM rows)

Detail App

(500 MM rows)

Cross Subject Navigation

Summary

Dashboard

(20 MM rows)

Products

Dashboard

(10 MM rows)

Customer

Dashboard

(5 MM rows)

Orders

Dashboard

(50 MM rows)

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


26

Application Architectures Hybrid

Many Big Data use cases can be addressed with a hybrid approach

In-Memory Dashboard

(Millions rows)

Drill-to-Detail

Direct Discovery Application

(Billions)

Historic Trends

Direct Discovery

Dashboard

In-Memory

Dashboard

Direct Discovery Application

(Billions)

Time Sensitive

Direct

Discovery

Application

(Billions)

In-Memory

Dashboard

(Millions)

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


27

Big Data – Achieving Performance

Hot

Warm

Cold

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


28

Hadoop - Connectivity

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


29

Hadoop – Query Performance

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


30

Tech Partner Ecosystem

Qlik & Big Data

Examples

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


32

Qlik Sense / Optimized Hadoop / JethroData

• 2.5 Billion Rows via Direct Discovery

• http://jethrodata.qlik.com

Big Data demos

QlikView / Parstream

• 50 Billion Rows via Direct Discovery in Amazon

Cloud

• https://youtu.be/xSFbCtV7__8

http://jethrodata.qlik.com/

https://youtu.be/xSFbCtV7__8

https://youtu.be/xSFbCtV7__8

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


33

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


34

93 million

daily active users

1085 million

daily game plays

€1.22 billion

on in-app

purchases

1.6B rows

of data per day in

Hadoop

211M rows

per day extracted

for analysis

Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


35

The final part of the story is

adding context and relevance

and delivering it to people

at the point of decision.

“ “


Guidelines


Green

RGB = 97 - 167 - 41

Grey

RGB = 95 – 96 - 98

Green 2

RGB = 76 - 140 - 43

Grey 2

RGB = 126 - 128 - 131

Secondary colours

Qlik colour palette



Grey 3

RGB = 159 - 161 - 164

Green 3

RGB = 196 - 214 - 0

Orange

RGB = 248 - 152 - 29

Yellow

RGB = 255 – 199 - 44

White

RGB = 255 - 255 - 0


Questions?

#VizYourWorld

Contact me directly:

Mischa van Werkhoven ([email protected])

@MischaDaniel

Web:

www.qlik.com

www.qlikcommunity.com

mailto:[email protected]

http://www.qlik.com/

http://www.qlikcommunity.com/

big data is only half the story - qlik & big data

Data & Analytics