10 commandments of bi on big data
TRANSCRIPT
What’s so special aboutBI on Big Data?
12.14.15
Shant Hovsepian@superdupershant
Unified Visual Analytics & BI Platform for Big Data.
11.13.15
Sushil ThomasCo-Founder / CEO
Presentation prepared for Kaiser Permanente
Scott MurphyAccount Executive
Presentation prepared for Data Driven NYC #42
1
What’s so special aboutBI on Big Data?
12.14.15
Shant Hovsepian@superdupershant
Unified Visual Analytics & BI Platform for Big Data.
11.13.15
Sushil ThomasCo-Founder / CEO
Presentation prepared for Kaiser Permanente
Scott MurphyAccount Executive
Presentation prepared for Data Driven NYC #42
2
Co-Founder & CTO
Came out of stealth mode in June and just announced our GA
product release.
Rapidly Growing and focused on the Fortune 2000
See lots of customer struggles with data, Big and Small
You don’t use previous generation architectures to store
Big Data so why use previous generation BI tools to analyze it?
Create businessvalue fromBig Data
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Data Driven NYC #42 12.14.15 3
– OUR FOUNDING VISION –
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Data Driven NYC #42 12.14.15 4
@BigDataBorat
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Data Driven NYC #42 12.14.15 5
#BigDataSeacrest
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Data Driven NYC #42 12.14.15 6
#BigDataMoses
-The 10 commandments ofBI ON BIG DATA-
Thou shalt notmove Big Data
7Data Driven NYC #42 12.14.15
Moving Big Data Is Expensive
On-Cluster BI is now possible
Push all the computation down close to the data
Careful having to extract data out to data marts & cubes
-Lots of native analysis engines out there, make sure your BI tools support them.-ODBC/JDBC connectors aren’t always enough.-
-Having to extract data out of the system is slow and defeats the purpose of having a specialized architecture.-Extracts and cubes in situ aren’t so bad as long as it’s not a required first step to analysis.-
-YARN, Mesos, have made it possible to run a BI server right next to the data.-The benefits of unified management, performance, workload management are just huge when the infrastructure is converged.-
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
8Data Driven NYC #42 12.14.15
Thou shalt not stealor violate corporate security policy
9Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Data Driven NYC #42 12.14.15 10
Security is Serious
-All the serious Big Data infrastructure vendors have implemented some form of security, your BI tool should support it.-BI software shouldn’t require re-implementing all the access control rules all over again. -RBAC – Role Based Access Control-Single Sign On especially for embedded use cases-
Thou shalt not payfor every user or megabyte
11Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Be wary of pricing models that penalize you for increased adoption
-We’ve seen Big Data deployments quadruple in size and adoption within a couple of months-Keep an eye out for licensing models that bill for users or data size, these too can grow much quicker than you can anticipate-
12Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Be wary of pricing models that penalize you for increased adoption
-We’ve seen Big Data deployments quadruple in size and adoption within a couple of months-Keep an eye out for licensing models that bill for users or data size, these too can grow much quicker than you can anticipate-
13Data Driven NYC #42 12.14.15
BIGDATANOT
BIG$$$$
Thou shalt covetthy neighbor’s visualizations
14Data Driven NYC #42 12.14.15
First Class Support for Collaboration
SHAREPUBLISH-Export to PDF or email is expected by everyone.-Publish to server to preserve interactivity instead of a static image.-Supporting source data updates after publishing is even better.-
-Preserve data lineage and how.-Network effects, github for BI clone and fork.-
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
15
Collaborative exploration is needed because in some cases no single person understands the entire data set.
Data Driven NYC #42 12.14.15
Thou shalt analyze thine datain its natural form
16Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
This is What Big Data Looks Like-Free form text-
17Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
This is What Big Data Looks Like-Free form text-Key Value Pairs-
18Data Driven NYC #42 12.14.15
8=FIX.4.2^A9=145^A35=D^A34=4^A49=ABC_DEFG01^A52=20090323-15:40:29^A56=CCG^A115=XYZ^A11=NF0542/03232009^A54=1^A38=100^A55=CVS^A40=1^A59=0^A47=A^A60=20090323-15:40:29^A21=1^A207=N^A10=139^A
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
This is What Big Data Looks Like-Free form text-Key Value Pairs-JSON / Semi-Structured-
19Data Driven NYC #42 12.14.15
8=FIX.4.2^A9=145^A35=D^A34=4^A49=ABC_DEFG01^A52=20090323-15:40:29^A56=CCG^A115=XYZ^A11=NF0542/03232009^A54=1^A38=100^A55=CVS^A40=1^A59=0^A47=A^A60=20090323-15:40:29^A21=1^A207=N^A10=139^A
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
This is What Big Data Looks Like-Free form text-Key Value Pairs-JSON / Semi-Structured-Tables-
20Data Driven NYC #42 12.14.15
8=FIX.4.2^A9=145^A35=D^A34=4^A49=ABC_DEFG01^A52=20090323-15:40:29^A56=CCG^A115=XYZ^A11=NF0542/03232009^A54=1^A38=100^A55=CVS^A40=1^A59=0^A47=A^A60=20090323-15:40:29^A21=1^A207=N^A10=139^A
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
21Data Driven NYC #42 12.14.15
Don’t let your BIsolution tell youotherwise.
Thou shalt not waitendlessly for thy results
22Data Driven NYC #42 12.14.15
No Surprise Here, Things Should Be Fast
Take Samples of the Data
Build anOLAP Cube
Create Temp Tables
-This works pretty well once you’ve got a good idea of what metrics matter.-Don’t get stuck with “cube first results later”.-Make sure your cubes can live on cluster or scale out easily.-
-This can be simple as fancy caching. Make sure some of tables can be intelligently reused.-Materialize complex expressions so we don’t have to recalculate them every time.-Store them on cluster where they belong.-
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
23
Tricks legacy BI tools use to achieve performance
Data Driven NYC #42 12.14.15
-Instant gratification though the results may not be correct initially.-How far down can the samples be pushed, need to cognizant of blocking operations. -
Thou shalt not buildreports but apps instead
24Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
What comes to mind when I say reports?
25Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
What comes to mind when I say reports?
-Traffic Report-
26Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
What comes to mind when I say reports?
-Traffic Report-Weather Report-
27Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
What comes to mind when I say reports?
-Traffic Report-Weather Report-Book Report-
28Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
What comes to mind when I say reports?
-Traffic Report-Weather Report-Book Report-Report Card-
29Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
What comes to mind when I say reports?
-Traffic Report-Weather Report-Book Report-Report Card-
30Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
What comes to mind when I say apps?
31Data Driven NYC #42 12.14.15
Visual Information Seeking Mantra
RAILs made web apps easy, BI Tool should do the same.
Async data from multiple sources.
Interact with Visual elements.
-Lots of native analysis engines out there, make sure your BI tools support them.-ODBC/JDBC connectors aren’t always enough.-
-Having to extract data out of the system is slow and defeats the purpose of having a specialized architecture.-Extracts and cubes in situ aren’t so bad as long as it’s not a required first step to analysis.-
-YARN, Mesos, have made it possible to run a BI server right next to the data.-The benefits of unified management, performance, workload management are just huge when the infrastructure is converged.-
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
32
“overview, zoom and filter, then details on demand
Data Driven NYC #42 12.14.15
Thou shalt useintelligent tools
33Data Driven NYC #42 12.14.15
“Smart” BI Tools will help the user out.
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
34
-Help with suggesting Vizs to create.-Built in search for everything.-Automatically maintaining models and caches the burden isn’t on the end user.-
Data Driven NYC #42 12.14.15
Thou shalt go beyondthe basics
35Data Driven NYC #42 12.14.15
You don’t ask the same questions of your Big Data?
Make some of that functionality is available in an easy to use manner.
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
36
Big Data is a gold mind of predictive and advanced analytics use cases.
Data Driven NYC #42 12.14.15
Thou shalt use Arcadia Data
37Data Driven NYC #42 12.14.15
Thou shalt use Arcadia DataJust kidding
38Data Driven NYC #42 12.14.15
Arcadia Data 2015. Proprietary and Confidential. Kaiser Permanente 11.09.15 3
Outline
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
Company Introduction
Solution Overview
Enterprise Features
Customer Use-Cases
39
Arcadia DataConvergedAnalyticsPlatform
arcadiadata.com
Data Driven NYC #42 12.14.15
Thank you.
40
12.14.15
Data Driven NYC #42 12.14.15