real time analytics hpe vertica meetup.pdf · vertica analytics platform fast boost performance by...
TRANSCRIPT
![Page 1: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/1.jpg)
Real Time Analytics
![Page 2: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/2.jpg)
![Page 3: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/3.jpg)
Vertica
– A SQL analytic engine
– Built for Speed, Scale and Efficiency
– Supports standard SQL
– Provides rich Analytic functionality and is extensible
– Integrates well with Big Data ecosystem tools
– Runs on premises, in the Cloud, and on Hadoop
![Page 4: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/4.jpg)
What's wrong with this picture?
– SQL ??
– Real-time Analytics ???
– Real-time, continuous load ?
– Real-time, very short response time ?
– Big Data ????
![Page 5: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/5.jpg)
Vertica – Does it scale ???
select GET_COMPLIANCE_STATUS();
![Page 6: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/6.jpg)
Vertica – Does it scale ???(not a fake, believe me…)
select GET_COMPLIANCE_STATUS();
GET_COMPLIANCE_STATUS
--------------------------------------------------------------------------------
Raw Data Size: 2.75PB +/- 0.30PB
License Size : 1.95PB
Utilization : 141%
Audit Time : 2016-09-27 23:59:29.367875+00
Compliance Status : ***** NOTICE OF LICENSE NON-COMPLIANCE *****
Continued use of this database is in violation of the current license agreement.
Maximum licensed raw data size: 1.95PB
Current raw data size: 2.75PB
License utilization: 141%
IMMEDIATE ACTION IS REQUIRED, PLEASE CONTACT VERTICA
![Page 7: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/7.jpg)
Vertica – Is it really fast ?
– Trillion Row Qlik-on-Vertica Dashboard
– https://www.youtube.com/watch?v=ZnMDeg8V2sg
![Page 8: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/8.jpg)
Vertica – Is it so simple ?
– HPE Vertica and Qlik Direct Discovery: A Technical Exploration
– https://community.dev.hpe.com/t5/Vertica-Knowledge-Base/HPE-Vertica-and-Qlik-Direct-Discovery-A-Technical-Exploration/ta-p/234332
![Page 9: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/9.jpg)
Vertica – Is it so simple ?
– No !
– HPE Vertica and Qlik Direct Discovery: A Technical Exploration
– Implementation Methods
– Fact and dimension tables in-memory. Most applications are created using this approach. However, this paper does not cover the all-in-memory option because it is not suitable for big data (such as a few billion rows of fact data) and requires too much memory.
– Fact and dimension tables in Direct Discovery (regular star schema).
– BFFT (big flat fact table) in Direct Discovery. There are no dimension tables with BFFT.
– Fact tables in Direct Discovery and dimensions in memory.
– Multiple fact tables in Direct Discovery. This is not generally recommended because of complex design considerations.
![Page 10: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/10.jpg)
Vertica @ Nimble Storage
10
![Page 11: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/11.jpg)
Changing the game with the Internet of (Powerful) Things
InfoSight
![Page 12: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/12.jpg)
Nimble Storage – Some metrics
– >7,500 customers
– millions of virtual objects under continuous monitoring
– collected per day
– Database Characteristics
– Raw Data : 550TB - Disk: 200 TB - On Nimble: 100 TB
– 350K selects per day
– 60K inserts/deletes per day
– Configuration
– 2 Vertica clusters – 2x8 servers – 2x8x54 cores – Nimble Storage instead of DAS
>250 billion sensor values
>2 billion log events
>100 million configuration variables
![Page 13: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/13.jpg)
More on Vertica by Nimble Storage
– https://my.vertica.com/wp-content/uploads/2016/09/B10823_10823_Presentation_2.pdf
– From Vertica Big Data Conference 2016 : https://my.vertica.com/big-data-conference-2016/
![Page 14: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/14.jpg)
Vertica @ Criteo
14
![Page 15: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/15.jpg)
Hadoop for Primary Storage
and MapReduce
Cascading, Scalding and
Hive for Data Transformation
Hive and Vertica for
Data Warehousing
Tableau and ROLAP Cube
for Structured Data Access
Vizatra for speed
The analytics stack at Criteo
![Page 16: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/16.jpg)
More on Vizatra+Vertica by Criteo
–SBTB FinagleCon 2015: Justin Coffey, Presenting Vizatra – YouTube
–https://www.youtube.com/watch?v=uXmEhSFzNLs
![Page 17: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/17.jpg)
More on Vertica
![Page 18: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/18.jpg)
Vertica analytics platform
Fast
Boost performance by 500% or more
Scalable
Handles huge workloads at high speeds
Standard
No need to learn new languages or add complexity
Costs
Significantly lower cost over legacy platforms
18
![Page 19: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/19.jpg)
About Vertica
Massively Parallel Processing
– Shared Nothing
– Elastic scale-out architecture
– Built-in high availability
– Commodity Hardware
– Easy setup and administration
– And more …
Client Network
Private Data Network
20 TB 20 TB 20 TB
Node 1 2 x 12 Cores 128+GB RAM
Node 2 2 x 12 Cores 128+GB RAM
Node 3 2 x 12 Cores 128+GB RAM
![Page 20: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/20.jpg)
Core Vertica TechnologyBuilt for performance and scale
20
![Page 21: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/21.jpg)
my.vertica.com
–Download Vertica Community Edition on my.vertica.com
–Up to 1 TB and 3 nodes
21
![Page 22: Real Time Analytics HPE Vertica Meetup.pdf · Vertica analytics platform Fast Boost performance by 500% or more Scalable Handles huge workloads at high speeds Standard No need to](https://reader036.vdocuments.us/reader036/viewer/2022071218/60529bc1119fcb175f3f8cb0/html5/thumbnails/22.jpg)