big data made easy in the era of the cloud - demi ben-ari
TRANSCRIPT
![Page 1: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/1.jpg)
Big Data made easy in the era of CloudDemi Ben-Ari - VP R&D @ Panorays
![Page 2: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/2.jpg)
About Me
Demi Ben-Ari, Co-Founder & VP R&D @ Panorays● Google Developer Expert● Co-Founder of Communities:
○ “Big Things” - Big Data, Data Science, DevOps○ Google Developer Group Cloud○ Ofek Alumni Association
In the Past:● Sr. Data Engineer - Windward● Team Leader & Sr. Java Software Engineer,
Missile defence and Alert System - “Ofek” – IAF
![Page 3: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/3.jpg)
AutomatetheSecurityManagementofThirdParties
Capturethe Hacker’sView
GetRealtime Ratings
Complywith Regulations
![Page 4: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/4.jpg)
Say “Distributed”, Say “Big Data”,Say….
![Page 5: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/5.jpg)
What is Big Data (IMHO)? And What to Monitor?
● Systems involving the “3 Vs”:What are the right questions we want to ask?○ Volume - How much?○ Velocity - How fast?○ Variety - What kind? (Difference)
![Page 6: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/6.jpg)
What had happened in the last years?
● Storage got cheaper● The capacity of Data grew exponentially● Cloud service providers grew rapidly● Connectivity got much easier● Cloud made “by demand” computation possible● “Compute” started moving to the “Data” and not the other way.
![Page 7: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/7.jpg)
Situations & Problems
https://imgflip.com/i/1ap5krhttp://kingofwallpapers.com/otter/otter-004.jpg
![Page 8: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/8.jpg)
MongoDB + Spark
Worker 1Worker 2
….….
……
Worker N
Spark Cluster
Master
WriteRead
MasterSharded MongoDB
Replica Set
![Page 9: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/9.jpg)
Cassandra + Spark
Worker 1Worker 2
….….
……
Worker N
Cassandra Cluster
Spark Cluster
WriteRead
![Page 10: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/10.jpg)
Cassandra + Serving
Cassandra Cluster
WriteRead
UI ClientUI Client
UI ClientUI Client
Web ServiceWeb
ServiceWeb ServiceWeb
Service
![Page 11: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/11.jpg)
Distributed Microservices Architecture
Service A
Queue
DB
Service B
DBCache
Cache DBService C
Web Server
DB
Analytics Cluster
Master
Slave Slave Slave
Monitoring System???
![Page 12: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/12.jpg)
Did someone say Containers?
![Page 13: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/13.jpg)
Docker Environments
● Docker?
● Orchestration?
VS
● Wait, What about local mode? ○ Minikube vs Docker Engine
![Page 14: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/14.jpg)
Problems
● Multiple physical servers● Multiple logical services● Want Scaling => More Servers
![Page 15: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/15.jpg)
Data flow and Environment(Use Case)
![Page 16: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/16.jpg)
Structure of the Data
● Maritime Analytics Platform● Geo Locations + Metadata ● Arriving over time● Different types of messages being reported by satellites ● Encoded (For compression reasons)● Might arrive later than actually transmitted
![Page 17: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/17.jpg)
Data Flow Diagram
External Data
Source
Analytics Layers
Data Pipeline
Parsed Raw
Entity Resolution Process
Building insightson top of the entities
Data Output Layer
Anomaly Detection
Trends
UI for End Users
![Page 18: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/18.jpg)
Environment Description
Cluster
Dev Testing Live Staging ProductionEnv
OB1K
RESTful Java Services
![Page 19: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/19.jpg)
Monitoring Your Data
https://memegenerator.net/instance/53617544
![Page 20: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/20.jpg)
Data Questions? What should be measure
● Did all of the computation occur?
○ Are there any data layers missing?● How much data do we have? (Volume)
● Is all of the data in the Database?
● Data Quality Assurance
![Page 21: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/21.jpg)
Conclusions
● Keep all of the Data that you can● In its most raw form
● Duplicating Data is not a bad thing● By demand compute with save you much time and money● Find the relevant tool to solve each problem
● Not one tool that will solve all of them (No such thing)● Use the cloud as an auxiliary tool
● Will boost your productivity by much
![Page 22: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/22.jpg)
Questions?
![Page 23: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/23.jpg)
● LinkedIn● Twitter: @demibenari● Blog: http://
progexc.blogspot.com/● [email protected]
● “Big Things” CommunityMeetup, YouTube, Facebook, Twitter● GDG Cloud
![Page 24: Big Data made easy in the era of the Cloud - Demi Ben-Ari](https://reader031.vdocuments.us/reader031/viewer/2022030317/5a65c6957f8b9af2638b4cb3/html5/thumbnails/24.jpg)