big data ecosystem - 1000 simulated drones
TRANSCRIPT
![Page 1: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/1.jpg)
Big DataBig Dataecosystemecosystem
![Page 2: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/2.jpg)
Presentation about Espeo's BigData ecosystem based on 1000simulated drones flying around
Poznan city
![Page 3: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/3.jpg)
Drones produce and collect realtime sample data like:
latitude,longitude,height(m),temp(C),wind(m/s),humidity,air-polution
![Page 4: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/4.jpg)
![Page 5: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/5.jpg)
Drone soft written in Scala language
![Page 6: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/6.jpg)
Drone soft written in Scala languageIn real time it streams data to the server
![Page 7: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/7.jpg)
Drone soft written in Scala languageIn real time it streams data to the serverUsing Kafka
![Page 8: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/8.jpg)
Drone soft written in Scala languageIn real time it streams data to the serverUsing Kafka
![Page 9: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/9.jpg)
On a server, data is read by Spark Streaming.
It allows us to:
![Page 10: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/10.jpg)
It allows us to:
save data to Cassandrasend calculated data to browser through websocketsend it to another Kafka consumersave the whole log to Hadoop cluster
On a server, data is read by Spark Streaming.
![Page 11: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/11.jpg)
By saving logs to Hadoop cluster, we can lateraccess those logs, if we didn't save somethingin Cassandra
![Page 12: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/12.jpg)
By sending data to the browser throughwebsocket, we can see where our drones are inrealtime, monitor sensors and much more
![Page 13: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/13.jpg)
![Page 14: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/14.jpg)
By using Cassandra and Apache Spark datascientists can analyze given data later,by using:
1. Apache Zeppelin - Apache Spark(df, RDD) + Scala - Apache Spark MLLib2. Azure Machine Learning
![Page 15: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/15.jpg)
We prefer to use Azure Machine Learninginstead Spark MLLib because it is much easierto understand - and design new predictions
Read our blog post about Azure ML:http://espeo.eu/blog/azure-machine-learning-predictions/
![Page 16: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/16.jpg)
CompleteCompleteecosystem diagramecosystem diagram
![Page 17: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/17.jpg)
Drones
Wifi
![Page 18: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/18.jpg)
Drones
Wifi
![Page 19: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/19.jpg)
Drones
Wifi
websocket
![Page 20: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/20.jpg)
Drones
Wifi
websocket
![Page 21: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/21.jpg)
Drones
Wifi
websocket
![Page 22: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/22.jpg)
Drones
Wifi
websocket
API
![Page 23: Big Data Ecosystem - 1000 Simulated Drones](https://reader031.vdocuments.us/reader031/viewer/2022030222/58843b8c1a28ab39538b7577/html5/thumbnails/23.jpg)
Drones
Wifi
websocket
API