Download - BigData in IoT #iotconfua
![Page 1: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/1.jpg)
[Big]Data in IoT:from Lambda architecture to predictive maintenance
by Tatyana Matvienko
![Page 2: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/2.jpg)
IoT… Haven’t heard?
![Page 3: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/3.jpg)
The Internet of Things is...
● Healthcare
● Energy delivery (water, oil & gas)
● Connected city
● Smart vehicles (cars, elevators...)
● Security sensors and devices
● Monitoring and analytic systems
● Smart...
![Page 4: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/4.jpg)
The problemDevice
● Electronics knowledge
● Master-Slave architecture
● OSI model● Binary protocol● Cloud connectivity
Server
● Meta- and time-series data
● Data storage● Business logic● Data analysis● Client applications
Orchestration
● Scalability● Fault-tolerance● Administration● Responsibility
![Page 5: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/5.jpg)
SolutionDeviceHive
Machine to Machine (M2M) open source Communication Framework
![Page 6: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/6.jpg)
What is DeviceHive?
Firmware
● Install Ubuntu Snappy Core on your device
● BLE support● Firmwares● Gateways
Device Server
Java and .NET based servers
● REST & Websockets● API libraries in
different languages (JS, C, C#, Python...)
● Lambda-architecture
Orchestration
Simplify deployment as much as possible
● Documentation● Playgrounds● Wrap the containers
![Page 7: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/7.jpg)
REST/WebsocketsBINARY BINARY
Gateway Gateway
Device management
and monitoringData analysis and
visualisationBigData StorageReal time data
processing
![Page 8: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/8.jpg)
What it used to be...● JavaEE, EJB, Glassfish● Hibernate with RDBMS for meta and operational data
(PostgreSQL)● REST and Websockets handled the same way● Hazelcast for both - caching and messaging ● Integration with Docker● Admin console with the latest 100 notifications in the table (JSON)
![Page 9: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/9.jpg)
Step 1: Java server
![Page 10: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/10.jpg)
One day your IoT project becomes BigData challenge...
![Page 11: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/11.jpg)
Lambda-architecture ○ Batch Layer○ Serving Layer○ Stream Layer
![Page 12: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/12.jpg)
Lambda-architecture in DeviceHiveLambda architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch- and stream-processing methods.
![Page 13: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/13.jpg)
What was done...
● Message bus added (Kafka)● Only metadata in RDBMS● Operational data to message bus and cache only● REST endpoints are served from cache only● Cassandra worker streaming data from Kafka to C*● Moved to Spring and Spring boot
![Page 14: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/14.jpg)
Step 2: Deployment
![Page 15: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/15.jpg)
So, let’s setup our project...
● Download and build sources● Setup database● Configure Glassfish● Configure Kafka● Configure Zookeeper● Configure Cassandra (if needed
of course...)● Run● Fail
![Page 16: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/16.jpg)
Welcome to Juju
![Page 17: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/17.jpg)
Mesos & Marathon
![Page 18: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/18.jpg)
PintostackContainerization
● Every service is a container● Container is more than just an app, it is an environment● Container is a final build artifact● Containers for HDFS nodes, Databases, Microservices, Web Applications● Docker Containers are the first class citizens
Key Components● Infrastructure: supply of resources● Containers● Resource abstraction● Scheduling● Service discovery● Logging
![Page 19: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/19.jpg)
Step 3: Analysis and visualisation
![Page 20: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/20.jpg)
Freeboard.io
● A web-based tool● Fully customizable
and interactive user-interfaces
● Real time dashboards
![Page 21: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/21.jpg)
ELK (Elasticsearch + Logstash + Kibana)Open source, scalable solution to search, analyze, and visualize your data, allowing you to get actionable insight in real time
![Page 22: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/22.jpg)
Spark Streaming + Zeppelin
![Page 23: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/23.jpg)
Apache Spark
![Page 24: BigData in IoT #iotconfua](https://reader034.vdocuments.us/reader034/viewer/2022042611/58f0c9321a28ab17188b4679/html5/thumbnails/24.jpg)
Q&A?...